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We consider estimating the predictive density under Kullback- 
Leibler loss in an lo sparse Gaussian sequence model. Explicit expres- 
sions of the first order minimax risk along with its exact constant, 
asymptotically least favorable priors and optimal predictive density 
estimates are derived. Compared to the sparse recovery results in- 
volving point estimation of the normal mean, new decision theoretic 
phenomena are seen here. Sub-optimal performance of the class of 
plug-in density estimates reflects the predictive nature of the prob- 
lem and optimal strategies need diversification of the future risk. We 
find that minimax optimal strategies lie outside the Gaussian family 
but can be constructed with threshold predictive density estimates. 
Novel minimax techniques involving simultaneous calibration of the 
sparsity adjustment and the risk diversification mechanisms are used 
to design optimal predictive density estimates. 

1. Introduction and main result. 

1.1. Estimating the high- dimensional Gaussian Predictive Density. 

Background: The objective of statistical prediction analysis is to choose a 
probability distribution which will be good in predicting the behavior of fu- 
ture samples (Aitchison and Dunsmore, 1975). If the observed past data X 
and the unobserved future data Y are generated from a joint density /(x, y), 
the objective is to estimate the future conditional density of /(y |X = x), 
also referred to as the predictive density (Geisser, 1971). For practical pur- 
poses, we usually need to forecast functionals of the predictive density. Good 
predictive performances can be ensured by using functions of predictive den- 
sity estimates which are optimally chosen based on appropriate goodness of 
fit measure. 

Here, we consider flexible parametric predictive models outlined in Geisser 
(1993) which can accommodate most dependencies in the data. We ob- 
serve X with Xi , X2 , • • • , X mi independently generated from a parametric 
density /(e i0i )(-) indexed by unknown parameters and known parameters 



AMS 2000 subject classifications: Primary 62C20; Secondary 62M20, 60G25, 9IG70 
Keywords and phrases: predictive density, risk diversification, minimax, sparsity, high 
dimensional, mutual information, plug-in risk, thresholding 

1 



2 



A = {ai : 1 < i < mi}. The future data Y = {Yi, • • • , Y m , 2 } are generated 
from successive independent parametric densities {f(o.bi)(') '■ 1 < * < m 2} 
with invariant (over time) unknown parameters and known parameters 
B = {bi : 1 < i < TO2}. In such a predictive model the dependence between 
the past and the future is based on the time- invariant parameters 9. 

Generic Parametric Predictive Model: M 

PAST OBSERVATIONS: X; '^5' f {0; a . )(•) , i = 1, ■ ■ ■ ,mi 

FUTURE OBSERVATIONS: Y,- '~' / ( e , b 3 ) (•) , j = 1, ■ ■ ■ , roa. 

If is fixed the true predictive density of Y would be f(o,B)(') = TYJ^i f{e,bj)(')- 
We would like to estimate it by density estimates p(.|X = x). We use the in- 
formation theoretic measure of Kullback and Leibler (1951) as the goodness 
of fit measure between the true and estimated distributions 

L(0,*K-N) = /Wy) dy - 

Averaging over the past observations X, the predictive risk of the density 
estimate p(- |X = x) at 6 is given by 

(!) P{0,p)=ll / (M )(x) / (0 ,B)(y)log feyffi ) dydx - 

The relative entropy predictive risk p ( 0, p ) measures the exponential rate 
of divergence of the joint likelihood ratio over a large number of independent 
trials (Larimore, 1983). In classical fixed-dimensional parametric analysis, 
the minimal predictive risk estimate would maximize the expected growth 
rate in repeated investment scenarios (Cover and Thomas, 1991, Chapter 6 
and 15). Competitive optimal predictive schemes (Bell and Cover, 1980) for 
gambling, sports betting, portfolio selection, etc can be constructed from 
predictive density estimates with optimal Kullback-Leibler (KL) risk prop- 
erties. In data compression set-up L (0, p ( ■ x)) reflects the excess average 
code length that we need if we use the conditional density estimate p instead 
of the true density to construct a uniquely decodable code for the data Y 
given the past x (McMillan, 1956). The notion can be extended to a sequen- 
tial framework where minimizing the predictive risk would result in the min- 
imum description length (Barron, Rissanen and Yu, 1998, Rissanen, 1984) 
based estimate of the true parametric density (Liang and Barron, 2005). 
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The modern data deluge has influenced a rapid evolution of statistical 
methods towards simultaneous multi-parametric analyses (Donoho, 2000). 
The traditional fixed dimensional predictive density estimation problem 
needs extension to high-dimensional parametric models in the following ap- 
plications: 

Data compression: Coding of high-dimensional data (Candes, 2006, Liang and Barron, 
2004) needs construction of a decodable code for Y given the value 
of X. If the high-dimensional parameter is known the optimal ex- 
pected length of such a code would have been based on the true den- 
sity f(e,B)- I n universal data compression (Rissanen, 1984), without 
any prior knowledge of 6, a choice of predictive density p(Y|x) will 
be used instead of the true density to construct the code. The excess 
average code length in that case is given by: 

(2) K e [log 2 (i/?(Y|x)) - log 2 (V/ (e , s) (Y))] = L(0,p(.|x))/log2 bits. 

We ignore the issue of discretization as the loss will be the limit redun- 
dancy based on infinitesimally fine choice of discretizations (Csiszar, 
1973, Vajda, 2002). Now, if the parameter 9 was generated from the 
distribution ir then the minimal excess average code length is given 
by the Bayes risk of the prior ir on 6 (Liang, 2002). The integrated 
Bayes risk of the estimate p for the prior ir is given by B(ir,p) = 
f 7r(0)p ( 6, dO and the Bayes risk B(tt) of the prior equals min^ B(-K,p). 
It is equal to the mutual conditional information 1^(0; Y|X) between 
the unknown parameter 6 and the future data Y given the past X 
(see Appendix section A.l for the results in the case of Gaussian 
channels). Intuitively, based on the decomposition of /^(O; Y|X) = 
/^(QjX, Y) — /^(QjX) it can be seen that minimizing the Bayes pre- 
dictive risk B(ir,p) would signify extracting the maximum possible 
dependence of Y and based on X. The information capacity of 
the channel C is given by the maximal mutual conditional informa- 
tion max^-g^v! /^(G; Y|X) over an appropriate class of priors. Here, 
we will evaluate C by explicitly calculating the maximal Bayes risk 
max^g^ B(tt). 

Sequential Investment with side information: Investment schemes based 
on high-frequency trading need predictive strategies on financial in- 
struments governed by a large number of parameters (Fan, Lv and Qi, 
2011). The log-optimal predictive strategies of Barron and Cover (1988) 
will depend on the high-dimensional density estimates minimizing the 
predictive risk in Equation [1]. 
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Sports Betting: Online betting portals have not only increased traffic but 
also caused a massive transformation of the fixed-odds sports betting 
market (Buchdahl, 2003). Betfair , one of the leading betting ex- 
changes in U.K. matches 15 times as many daily transactions as the 
London Stock Exchange. These online stochastic markets allow bets 
with the most lucrative odds to be placed on the joint occurrences 
of several events (multiple bets). As historical data can be accessed 
through portal-supplied application programming interface, statistical 
techniques are being increasingly used in designing betting strategies 
(Magee, 2011) and multi-parametric models are required to estimate 
the multiple-bets probabilities. 

Often, in high dimensional models transformations of the data approximate 
to normality are available. So henceforth we impose Gaussian distribution 
assumptions and in particular consider prediction in the location model: 

High Dimensional Gaussian Predictive Model 

M.l X ~ N(A9, a* I) and Y~N(B9, a} I) 

where A and B are m\ x n and mi X n data matrices with n being very large, 
dp and are the volatilities of the past X and the future Y. The location 
structure depend on the time-invariant unknown vector 9 of length n. When 
9 is dense (non-sparse), predictive density estimation in M.l has been dis- 
cussed in Mukherjee and Johnstone (2012b). Here, we further assume that 
9 is sparse with few non-zero coefficients. We impose an £q constraint on 
the parameter space: 

(3) G(n,s) = 1 6) € W 1 : + 0] < sj . 

This notion of sparsity is widely used in modeling highly interactive sys- 
tems (represented by a large number of related parameters) which are dom- 
inated by only few significant effects. Sparse models have been successfully 
employed in biological sciences (Tibshirani et al., 2002), engineering appli- 
cations (Donoho, 2006) and financial modeling (Brodiea et al., 2009). The 
predictive model M.l with £q constraint on the location structure can be 
used for sparse coding and for prediction in sparse networks. 

Here, the minimax risk calculations will be based on the orthogonal Gaus- 
sian model. Like sparse point estimation (Raskutti, Wainwright and Yu, 
2011, Zhang, 2010), constrained predictive density estimation in M.l would 
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intrinsically depend on risk calculations in the orthogonal model: 



Predictive Gaussian Sequence Model 



M.2 X~ N(0, all) and Y ~ N(0, a\ T) 



where X and Y are both n - dimensional vectors. M.2 is known as the 
homoscedastic Gaussian sequence model (Nussbaum, 1996) and has been 
widely studied in the function estimation framework (Johnstone, 2012). Op- 
timal estimation in M.l can be linked with the minimax decision theoretic 
results in M.2 through the procedure outlined in Donoho, Johnstone and Mont 



Sparse Gaussian predictive density estimation has the attributes of a 
sparse prediction problem adapted to the peculiarities of the entropy loss 
function. Point prediction analyses of dense signals in M.l (Dicker, 2012, 
Huber and Leeb, 2012, Leeb, 2009) relate the worst-case performance with 
the spectral distribution of the predictors. Here, we concentrate on the or- 
thogonal model. We address some of the unresolved issues associated with 
the role of sparsity in prediction theory. Next, we describe in more detail 
the minimax predictive density estimation problem in the sparse orthogonal 
model M.2. 

1.2. Minimax estimation under sparsity constraints. 

In order to help the reader to better understand the context and nature of 
the predictive problem, we provide a brief review of the literature around the 
predictive density estimation problem. Aitchison (1975), Murray (1977), Ng 
(1980) showed that in most parametric models there exist Bayes predictive 
density estimates which are decision theoretically better than the maximum 
likelihood plug-in estimate. As the name suggests, a plug-in or estimative 
density estimate f^g B ^ belongs to the same parametric family of the true 

density and has the point estimate plugged in the place of the unknown 
parameter. Given a prior ir over M n the Bayes predictive density in M (along 
with some mild conditions) minimizes the intergrated Bayes risk and is given 



(2011). 



by 




where the posterior distribution 



(5) vr(0 |x) = {m 7r (x)} 1 / (0)A )(x)7r(0) and m,(x) = / f( 0jA ) (x) 7r(0) dO 



6 



is the marginal distribution. An important issue in predictive inference has 
always been to compare the performance of the class £ of point estima- 
tion (PE) based plug-in density estimates (Barndorff-Nielsen and Cox, 1996) 
with that of the optimal predictive density estimate. In fixed dimensional 
parametric spaces, large sample attributes of the predictive risk of efficient 
plug-in and Bayes density estimates have been studied by Komaki (1996) 
and Hartigan (1998). These results are independent of specific distributional 
attributes of / (Asian, 2006) and reflect the predictive nature of the problem 
through the relative inefficiency of the maximum likelihood plug-in density 
estimates. 

Recently, the predictive density estimation problem has been studied in 
high dimensional parametric spaces (George, Liang and Xu, 2006, Ghosh, Mergel and Datta, 
2008, Komaki, 2004, Xu and Zhou, 2011). Decision theoretic parallels be- 
tween predictive density estimation under Kullback-Leibler loss and point es- 
timation under quadratic loss have been explored in M.2 (George, Liang and Xu, 
2012). Fundamental techniques and results in unconstrained Gaussian point 
estimation theory (Brown, 1971, Brown and Hwang, 1982, Stein, 1974, Strawderman, 
1971) can be extended to produce optimal predictive density estimates 
(Brown, George and Xu, 2008, Fourdrinier et al., 2011, Komaki, 2001). Ta- 
ble 1 summaries these decision theory results in the point estimation and 
predictive density estimation regimes. The Bayes predictive density from 
the uniform prior pjj is the best invariant as well as a minimax density esti- 
mate in the unconstrained parametric space. Its risk properties are similar 
to those of the canonical minimax point estimate X. Both regimes exhibit 
inadmissibility of the best invariant estimates in their respective domains 
and improved minimax estimators are constructed. 

Another important subclass of density estimates are Linear estimates (C) 
which are Bayes rules based on the conjugate product normal priors. The 
resultant density estimates pi[ ot] = YYi=i N(otiXi, aj+Oj), with ai € [0,1], 
are still Gaussian but has larger variance than the future density f e a 2 (y ) . 
We choose the name 'linear' because the conjugate prior implies linearity 
of the posterior mean in X. It should be noted that for linear estimates, 
shrinkage of the location estimate X is related to flattening of the variance 
and £ n C consists only the zero density estimate. 

Xu and Liang (2010) showed that the class C is minimax optimal if the pa- 
rameter space is restricted to ellipsoids with certain growth conditions. Here, 
we evaluate the minimax risk over the Iq sparse parameter space 0(n, s) in 
the asymptotic framework {n — > oo and s/n — > 0}. Sparse point estimation 
has been extensively studied in this asymptotic set-up by Donoho and Johnstone 
(1994a,b), Donoho et al. (1992), Foster and George (1994) and the results 
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are the building blocks for popular sparse estimation methods (Candes and Tao, 
2007, Donoho, Maleki and Montanari, 2011, Zhang, 2005). It is natural to 
look for parallels in the predictive density regime. 



Table 1 

Parallels between Point Estimation of the Normal mean under quadratic loss and 
Predictive Density Estimation under KL loss 



Decision Theoretic Issues 


Point Estimation 


Predictive Density 


Admissibility in 
Unrestricted space 


Shrinkage priors 


Stein (1974) 
Strawderman (1971) 


Komaki (2001) 

George, Liang and Xu (2006) 


Complete Class & Baycs Rules 


Brown and Hwang (1982) 


Brown, George and Xu (2008) 


Minimaxity over & 
Restricted space 


Ellipsoids 


Pinsker (1980) 


Xu and Liang (2010) 


Sparsity Constraints 


Donoho et al. (1992) 


Evaluated here 



Our contribution: Instead of parallels, we found contrasting results for 
sparse estimation in the two regimes. The asymptotic minimax predictive 
risk reflects the nature of the predictive density estimation problem through 
the ratio r of the future to the past volatilities r = a^jo^. As r decreases, 
we need to estimate the future observations based on increasingly noisy 
past observations and so, the difficulty of the density estimation problem 
also increases. Unlike point estimation, sharp decision theoretic rates in the 
predictive density problem should depend on r. This dependence was not 
emphasized in the admissibility results in the unrestricted space. 

In our £q sparse prediction framework, as the proportion of non-zero sig- 
nals goes to zero, we find that the order of the minimax rate does not depend 
on r. So, exact determination of the constants of the minimax risk is impor- 
tant here. Optimal minimax estimators can be constructed by incorporating 
the predictive nature of the problem through the notion of diversification 
of the future risk. Under sparsity constraints efficiency of the prediction 
schemes depend on careful coupling of the sparsity adjustment and the risk 
diversification mechanisms. The risk diversification notion can also be ex- 
tended (though not done here) to dense unrestricted parametric spaces where 
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future uncertainty can be effectively shared by optimally flattening proba- 
bility densities based on the quadratic risk estimate of their corresponding 
location point estimator. 

Here we also evaluate the minimax risk over the wide class Q of all product 
Gaussian density estimates pc[6, d] = T\i = i N(0i, di). Q contains both C 
and £ and would represent the infamily error rate of the sparse Gaussian 
predictive density estimation problem. We prove that the sub-class Q is 
sub-optimal rand provide asymptotic miximax strategies as well as the sub- 
optimality rates of the sub-classes £,£ and Q. 

1.3. Description of the main results. 

Notations and Preliminaries. To proceed further we need some notation. 
The action space A n contains all possible densities in W 1 . The n-dimensional 
minimax risk of the prediction problem is given by 

R(n,s,r) = min max p(9,p). 

p£A n 0SO(n,s) 

We compute the limiting behaviour of R{n, s, r) in the asymptotic framework 
T = {n — > oo, s/n — > 0}. The minimax risk over the sub-class of Plugin {£) 
density estimates is represented by 

R(n, s, r, £) = min max p (0,pe[0]) where pe[Q] = N(6, a 2 f l n ). 

§ 0e0(n,s) 1 

Similarly, the minimax risk over the sub-classes of Linear (£) and Gaussian 
density estimates (Q) will be denoted by R(n, s, r, C) and R(n, s, r, Q) re- 
spectively. The maximum Bayes risk over the class of priors M{n) on M n is 
denoted by 

B(r,M(n)) = max min B(tt,p). 

7rSA / l(n) p 

As defined in Subsection 1.1 this maximin value is also the information 
capacity. A prior maximizing this Bayes risk is said to be a least favorable 
prior for the prediction problem. We evaluate the supremum Bayes risk of 
the following class of priors 



M(n,s) 



^ i=i ^ 



Univariate Prediction Problem. In high dimensions, due to concentration 
of measure, the decision theoretic results in our multivariate set up J 7 will 
intrinsically depend on the properties of the coordinate-wise univariate risk. 
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The least favorable priors as well as the minimax density estimates will be 
product densities. So, computing the multivariate risk in the n-dimensional, 
s-sparse orthogonal Gaussian Model M.2(n, s, a p , aj) would involve study- 
ing the corresponding univariate model M.2(l, i], a p , Of) in which we relax 
the sparsity constraint to restriction on the univariate prior space 

m ( ?? ) = {vr e V(R) : vr(fl / 0) < r?} 

where 'P(M) is the collection of all probability measures in M. The maximin 
value sup^g,,,^) infp i?(-7r,p) of this univariate prediction game is given by 
the maximal Bayes risk /3(yi,r) := sup 7rgm ( r) ) B(ir). The minimax risk for 
this univariate prediction problem is given by 

(6) PAi(rii r ) ■= inf sup B(ir,p). 

P 7r£m(»7) 

The minimax risk and the maximal Bayes risk over univariate sub-collection 
m of priors of m(r?) are respectively denoted by pM(Vi r i m ) an d ,5(^7, r, m). 
When the maximal Bayes risk (maximin) equals to the minimax risk, it is 
referred as the Bayes-Minimax risk for the prediction problem. 

As in our asymptotic framework J- the proportion of non-zero signals 
s/n goes to zero, the univariate risk calculation will be in the asymptotic 
regime 77 — >• 0. The difference between the multivariate and univariate cases 
is notationally demonstrated through the bold representation of multivariate 
vectors. The other non-standard notations used are (j)(\6,r) for the multi- 
variate normal density with center 6 and covariance rl while <I> = 1 — 
with $ being the standard normal distribution. For sequences, the symbol 
o-n ~ b n means a n = b n {l + o (1)) and a n x b n means a n /b n G (ci, C2) where 
ci and C2 are constants. 

Results. Consider the following symmetric univariate prior 3-point prior 
k[v, r, 3] = (1 - ??) • S + - r) ■ S Vri + - 77 ■ 5- Vr) 
where is the positive root of the quadratic equation 
v^v 2 + 2v~ 1 / 2 va = \l 



with v w = (1+r 1 ) , a = max( y^og v w X 2 , 1) and Ag = 2a 2 log{(l — rj) r] x } 
is close to the universal threshold (when rj = n" 1 ) seen previously in PE 

— 1/2 

(Donoho and Johnstone, 1994a). As r] — > 0, the solution u v — > Xf = v w A e . 
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Also, consider the discrete cluster prior 7r[rj, r, CL] with 1 — rj probability 
at and sharing the remaining mass among a cluster of support points. 
The non-zero support points start from ± u„ and span out symmetrically on 
either side in a geometric progression with common ratio (1 + 2r) up to the 
universal threshold: 

K v 

(7) Trfo, r, CL] = (1 - V ) ■ 5 + ^L- ]T { 5^ + 5^ } where 

^ i=i 

(8) K v = max{i : (1 + 2r)''~ 1 ^ < A e + a}, 

(9) ^ = (1 + 2T) 1 - 1 i/,,, i = l,2,...,K v . 

For any fixed r £ (0, oo) as i) we have 

log(l + r ) 



A(r) = lim A' 
?7->0 ' 



21og(l + 2r) 



We will use the Bayes predictive density p (-\x; ir[r], r, CL]) derived from 
the cluster prior n[r],r, CL] to construct threshold estimates. Consider the 
following univariate threshold estimate which uses the best invariant den- 
sity estimate p(-\x;7rjj) from the uniform prior above the threshold A e and 
p (-\x; ir[rj, r, CL]) below the threshold 

do ^t,cl,u]( S |,)^{|*:*- c1 i> :;n^ : . 

The estimator p[rj, T, CL, U] attains the univariate minimax risk as rj — > 0. 

Theorem 1.1. For any fixed r £ (0, oo) as rj — >■ m me univariate 
prediction problem we have 

p M ( v ,r) = P(r l ,r) = 1 l^(l + o(l) 

Also, ir[n, r, 3] is cm asymptotically least favorable prior andp[rj, T, CL, U] is 
an asymptotically minimax optimal estimate. 

Based on the univariate version, we can construct a multivariate, co- 
ordinate wise rule 

n 

p[n, s, T, CL, U](y|x) = \[p[s/n, T, CL, V]{y t \ Xi ) 

i=l 
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which will be asymptotically minimax optimal in the high dimensional regime 
T . Also the product discrete distribution 

n 

ir[n,s,r,2,](6) = Hn[s/n,r,3](6i) 
i=i 

based on the 3-point prior will be asymptotically least favorable. The fol- 
lowing theorem, which is our main result, describes the minimaxity results 
for the predictive density estimation problem with (.q sparsity constraints in 
Model M.2. 

Theorem 1.2. As n — > oo, if s — > oo but s/n — > then for any fixed 
r E (0, oo] we have: 

a. The minimax risk R(n,s,r) ~ (1 + r)~ l s log(n/s). 

b. ir[n, s,r, 3] is an asymptotically least favorable prior distribution, i.e. 

B{ir[n, s, r, 3]) ~ sup inf B(ir,p) 

where V(@(n, s)) is the collection of all probability measures over 
e(n,s). 

c. The predictive density estimate p[n,s, T, CL, U] is minimax optimal, i.e. 

max p (6,p\n, s, T, CL, U]) ~ R (n, s, r). 

0£O(n,s) V ' 

We compute the multivariate minimax risk over the different sub-classes of 
predictive density estimates. As an immediate corollary of the above theorem 
it follows that the class of plug-in estimators £ is sub-optimal. The plug- 
in sup-optimality ratio R(n, s,r,£) / R{n, s,r) asymptotically equals 1 + r _1 
(see Lemma 6.1). As in point estimation, the class of linear estimates C 
performs very poorly. 

Lemma 1.1. For any fixed r G (0, oo) and for all seauences s n such that 
s n — > oo and s n /n — > as n — > oo, we have 

. R{n,s n ,r,C) 
lim ml r— = oo . 

n-»oo R(n,s n ,r) 

We also find that the performance of the wider class of all Gaussian density 
estimates is no better than that of plug-in estimates. 
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Lemma 1.2. Under the condition of Theorem 1.2 we have 

R(n,s,r,Q) ~ R(n,s,r,£). 

If the parametric space @(n,s) does not have any sparser representa- 
tion with respect to the group of orthogonal transformations then the sub- 
optimality of the class Q comprising of all Gaussian densities N(0, S ) in- 
cluding non-diagonal covariances is 1 + r _1 . 

Now, p [re, s, T, CL, U] is not derived from a prior and we would like to 
construct a prior for which asymptotic minimaxity in Theorem 1.2 holds. 
Consider another symmetric univariate prior n[r], r, INF] whose support con- 
sists of the origin and infinite number of equidistant clusters each containing 
2K V points (defined before in equation [7] ) in the same spatial alignment 
as for ir[n, r, CL]. As rj — > 0, the clusters centers are separated by A e and, as 
they move away from zero they have geometrically decaying probability with 
r\ being the common ratio. However, within clusters all support points are 
not equally likely any more. They have geometrically decaying probability 
with common ratio logr/ -1 . 

7r[r/, r, INF] = (1 - rj) ■ 6 Q + — ^ £ V +1 £ Qi [6^ + 5-^] where, 



2 

j=0 t=l 



oo; 



Hi = J A e + (1 + 2r) % u v , i = 1, . . . ,K V and j = 1, . . . 
qt = (logrT 1 )- 1 for i = 2,... ,K V and q x = 1 - 



log 7] 1 — 1 

Based on 7t[t/, r, INF] we can construct a multivariate prior 7r[n, s, r, INF](0) = 
niLi 7r [ s / n i r i INF](0j) in M n which will not only be least favorable but also 
yield a minimax optimal density estimate. As ir[n, s, r, INF] is a proper prior 
it is admissible. Though its support is not confined to 0(n, s) it concentrates 
on it asymptotically. It represents an equilibrium solution for the sparse min- 
imax prediction problem. 

Theorem 1.3. Under the conditions of Theorem 1.2 for any fixed r E 
(0, oo], the proper prior distribution ir[n, s,r, INF] is an asymptotically least 
favorable and its corresponding Bayes predictive density is asymptotically 
minimax optimal. 

These results reflect the predictive nature of the problem. The cluster 
prior 7r[?7, r, CL] in the minimax estimate p[rj, T, CL, U](y|x) diversifies the 
predictive risk over the constrained parametric space. The risk diversification 
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notion is essential to construction of optimal estimates and can be extended 
to other different forms of asymptotically minimax predictive density esti- 
mates. This mechanism of uncertainty sharing in presence of sparsity has not 
been previously described in minimax decision theory. To rigorously inter- 
pret the results, we need the risk equations in George, Liang and Xu (2006) 
which connect the Bayes predictive risk and with the square error of the 
posterior mean (see Section 2.1). Next, we provide an heuristic explanation 
of the implications of the results by an asymptotic (as r\ — > 0) risk analysis 
of univariate threshold estimators. 

New Phenomena in Estimation Theory. To adjust for high sparsity we 
use threshold based non-linear estimates t[X, S] with the threshold cut off 
at A, the best invariant estimate p(-\x;ttjj) above the threshold and esti- 
mate/scheme S below the threshold. We found that for such an estimate 
the threshold choice is dictated by the level of sparsity rj and can not be 
lower than A e . 

Lemma 1.3. For any fixed r £ (0,oo), scheme S and u S [0, 1), we have 

.. su Pirem(ij) B(ir, t[u\ e , S]) 

hm — = oo. 

r?->0 PM{V, r ) 

However, unlike PE, here the non-zero support point of the least favorable 
prior is not at A e but at Ay. So, the univariate asymptotic maximal Bayes 
risk P(t], r) ~ (2 r)~ l i] Aj is lower than the corresponding maximal quadratic 
Bayes risk f3 q {n,r) = n Ag after adjustment by 2r. Because of the threshold 
choice, the univariate threshold risk p(9,t[X e , ■]) is bounded when \9\ > A e . 
So, we need to restrict the predictive univariate risk in the region {9 6 
(— A e ,A e )} below the minimax risk. For that purpose, unlike PE we can 
not just use the zero estimator ^>(-|0, cr?) below the threshold because then 
p(9,t[X e ,0]) exceeds n~ l pM(v> r ) (given by equation [6]) in the region V = 
{9 : \9\ £ [A/,A e ]}. It leads to inefficiency of the optimal plug-in density 
estimates. 

Instead using the Bayes density estimate from Tr[q, r, 3] the risk can be 
controlled in the neighborhood around Xf but exceeds /3(?y, r) as 9 moves 
further away. The univariate threshold estimate t[X e , Tr[n, r, 3]] represents 
an unshared threshold prediction scheme. To control the risk through out 
we need to share the predictive risk for 9 between Xf and A e . The cluster 
prior serves the purpose by using a prior with probability 1 — n at (which 
controls the risk at 0) and distributing the remaining mass n equally among 
a finite chain of points covering V . We also find that discreteness of the 
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Plug-in Scheme 



Unshared Prediction 



Diversified Prediction 



Bounded risk 



Fig 1. Schematic diagram of KL-risk functions for different Predictive Schemes. As the 
true parameter 8 varies, the univariate asymptotic predictive risk lim l; _ ! .o p(6,t[X e , S]) is 
represented on the ordinate. The blue box between A/ and A e represents a support point of 
the cluster prior (representative of shared predictive schemes) which is not in the support 
of ir[n,r, 3] or other unshared predictive schemes. 



sharing scheme is important and continuous uniform sharing scheme will 
not work here. Also, the number of support points in the sharing scheme 
is proportional to r~ l reflecting the increasing difficulty of the prediction 
problem. Table 2 shows the number of support points in the cluster prior 
as r varies. In Figure [1], we have a schematic description of the asymptotic 
(i] — > 0) behavior of p(0, t[X e , S]) for different type of schemes in S. 



Table 2 

Number (K^) of positive support points in the cluster prior 7r[r],r, CL] as r varies. 



r 


0.1073 


0.1235 


0.1465 


0.1826 


0.2485 


0.4196 


> 0.4196 


K n 


7 


6 


5 


4 


3 


2 


1 



1.4. Organization of the paper. The proof of the results along with their 
implications are developed first in an overview fashion in Section 2 which 
may suffice for a first reading. Along with the general proof strategies it 
contains the proof of Theorem 1.1. Part of the proof of Theorem 1.2 extends 
over to Section 5. Technical proofs of all the statements in Section 2 are 
presented in Section 3 and Section 4 with some of the lemmas pushed to 
the Appendix to improve flow and readability. Theorem 1.3, Lemma 1.1, 
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Lemma 1.2 and Lemma 1.3 are proved in Section 6. The proofs involving 
direct risk calculations with the KL loss may be of independent interest and 
use in information theory. 

2. Proof overview and interpretation of the results. 

Hereon we will assume that a^ = l and a* = r. The general predictive KL 
risk as well as the £q constraints on the parametric space will not be af- 
fected by this restriction. However, the density estimates are usually based 
on statistics equivariant to the scale transformation and needs multiplica- 
tion by a p . The proofs as well as the interpretation which will be mostly 
done on the reduced univariate model are presented for the case ap 1 = 1 and 
er? = r. While extending the results to the multivariate set-up we will appro- 
prately modify the estimators for general o~ p and oy. Proper interpretation 
of the predictive results will involve comparison with quadratic risk of point 
estimation in model M.2(l, n, r). Next, we describe the connections for the 
univariate version. 

2.1. Relations with Point Estimation: Connecting Equations and path of 
experiments. 

The Kullback-Leibler (KL) predictive risk is connected to risk calculations in 
point estimation (PE) theory via the semi-futuristic random variable W = 
v w (X + r~ l Y) where v w = (1 + r -1 )" 1 . W would have been the UMVUE for 
the unknown location parameter if the future Y were also known along with 
the past X. The connections between the predictive and point estimation 
theory center around the parallel to the Tweedie's formula (Brown, 1971, 
Efron, 2011, Robbins, 1956) which gives a cloased from expression of the 
Bayes estimate n corresponding to prior tt for the location estimation 
problem under quadratic loss 

(11) n (X) =X + Vlo gm7T (X,l) 

where m n (Z,v) = J 4>(Z \0,v)tt(0) d0 denote the marginal distribution of 
a Gaussian random variable Z with variance v and with prior distribution 
tt on the location parameter 0. Bayes predictive densities for KL loss in 
M. 2(1, 77, r) is analogously related to the best invariant density estimate p\j: 

(12) p n (Y\X = x) = {m w (W x ,v v ,)m- 1 (x,l)} xp v (Y\X = x) 

where W x = v w {x + r~ l Y). As such, the Bayes risk in these two regimes are 
also related. By Brown, George and Xu (2008, Theorem 1) the predictive 
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risk of any prior 7r in M.2(l, rj, r) with a 2 = 1 is given by m^iz; 1) < oo for 
all z E M we have, 

If 1 

(13) p{9,%) = - v- 2 q{6,e^v)dv 

where q{9,9 1T ,v) denotes the quadratic risk E \\ 9^ — 9\\ 2 based on the 
general model M.2(l, rj, r) with a 2 = v. Through the connecting equations 
the Kullback-Leibler risk can be viewed as an weighted aggregation of the 
square error risk. 




Fig 2. Path of Experiments:- In red we have the true density of the observed random 
variable X around the unknown location 9. In different shades of gray (dark to light) 
we have respectively the density of Y a for a equal 0.10, 0.20, 0.33, 0.50, 0.67 and 1.00. 
Hence the corresponding v equal 1.00, 0.98, 0.95, 0.89, 0.82 and 0.67. When, a = 1, Y a 
corresponds to the true future density around 9 with known future variability r — 2. 

As the variance of Z varies from 1 to v w = (l+r -1 ) -1 it marks the gradual 
assimilation of the information in future Y to the existing information about 
9 in X through a path of memoryless experiments conducted separately for 
a £ [0, 1]. At each stage a along the path we observe (X, Y[a) ) where Y[a] 
is a Gaussian random variable around the true unknown location 9 and 
with variability r • a~ 2 . Along the path, the information about the unknown 
location 9 percolates through the sufficient statistics {Z[a\ : a G [0,1]} 
where Z[a] is the UMVUE, of 9 based on observing (X, Y[a\). As a increases 
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in [0, 1], v decreases from 1 to v w and we have, 

(14) Z[a] = X M + q2 A^[q] where a = r 1/2^-1 _ ^1/2 G [Q) jj and 

1 + a r 

(15) 81 = Z[a}+vVlogm n (Z[a},v) 

By equation [27] (in the appendix) the plug-in risk p v E (0, 0) equals q(9, 9 n ,v)/ (2v) 
and we see that p(9,p w ) is equal to f v~ 1 p E (6, 6^) dv which implies that 
the predictive KL risk is a linearly weighted (according to precision) accumu- 
lation of the corresponding plug-in risk. Using the connecting equation [13] 
most of the calculations involving risks of different Bayes predictive densities 
would follow easily by using known evaluation of the quadratic risk of the 
corresponding posterior mean. 

2.2. Bayes-Minimax Method. 

We will explicitly solve for the equilibrium of the univariate minimax prob- 
lem in M.2(l, 77, r). Using the minimax theorem here, we see that over the 
class 1x1(77) the maximal univariate Bayes risk f3(r],r) = sup{i?(7r) : ir G 
xn(r])} is always less than the minimax risk pM(r],r). So, if we can produce: 

1. a lower bound on (3{r],r) by considering the Bayes risk of a particular 
prior 7T (say); 

2. an upper bound on the minimax risk pM(i],r) by considering the 
maxfl p(0,po) for a particular estimator poj 

3. such that the lower bound and upper bound matches asymptotically 
as 77 — > 0; 

we can conclude that (3(tto) is the supremum Bayes risk as well as the min- 
imax risk and ttq is asymptotically least favorable and po is a minimax 
strategy for the univariate predictive density estimation problem. 

Once we have found the equilibrium for the univariate game, we extend 
the solution to the multivariate regime by following the general strategy out- 
lined in Johnstone (2012, Section 4.11). Considering the class of exchange- 
able priors m e we can reduce the n-dimensional multivariate problem as re- 
peated (n times) independent playing of the univariate minimax game and 
using the minimax theorem A.l we can show that R(n,s,r) would be less 
than n/3(s/n,r) (see Lemma 5.1). As detailed in Section 5, we can actually 
show that R(n, s, r) ~ n/3(s/n, r) would follow from concentration properties 
of the multivariate least favorable prior ir[n, s, r, 3]. Following this scheme, 
we now calculate the asymptotic univariate minimax risk. 
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2.3. The univariate asymptotic set-up. 

Here onwards we would further restrict our univariate parametric space to 
the non- negative orthant. The corresponding prior space would be m + (r]) = 
{ir(9) : 7r(0) > 1 — rj]. It would simplify exposition and the results easily 
generalizes over m(?7) by symmetrization. 

To produce a lower bound on the maximal Bayes risk, we consider the 
class of all 2-point priors in m + (r/). We will see that the 2-point version of 
the prior ir[rj, r, 3] will attain the maximal Bayes risk where 



where v w = (1 + r 1 ) 1 is the variance of the semi- futuristic random variable 
If and a = (2 log A/) 1 / 2 and X 2 f = 2v w log{(l - rj) r/" 1 }. 

To provide a detailed description of the univariate asymptotic regime we 
describe some fundamental quantities (functions of the sparsity level 77) as- 
sociated with our asymptotic calculations: 

Universal Threshold: A e = (21og{(l — 77) t? -1 }) ^\ is the universal threshold 
for point estimation of 9 based on the past X only Donoho and Johnstone 
(1994a). The subscript 'e' emphasizes its estimative purpose. Later on, we 
will see that A e is also the optimal threshold in the predictive regime. 
Ideal Predictive Threshold: \f = (2v w log(l — n) rj' 1 ) 1 / 2 ] is the universal 
threshold needed to devise minimax optimal threshold point estimate of 
9 based on observing the random variable W i.e both the past X and the 
complete future Y (which is equivalent to observing Y a with a = 1). The 
subscript / reflects its dependence on the future data. 
Vantage Point: u„ - the positive root of Equation [16] is the non- zero support 
point of the asymptotically least favorable 2-point prior in point estimation 
of 9 under quadratic loss and noise variability v w (compared with Johnstone 
(2012, Equation (8.36))) which again corresponds to location point esti- 
mation based on W. u„ will be pivotal to our calculations. u„ marks the 
beginning of the Vulnerable Zone which spans from [z/„, A/]. The calculation 
of the predictive risk on either side on u„ displays the sparsity adjustments 
and uncertainty sharing dynamics. 

Resolution Parameter, a = (2 log Aj) 1 / 2 : As rj — > 0, we need to compute the 
asymptotic predictive risk as the true parameter 9 moves along the non- 
negative axis. In the asymptotic regime, we can exactly quantify the risk 




and u v is the positive root of the quadratic equation 
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except at a few transition points (may be). However, the discontinuity of our 
analysis will only be limited to O (a)-neighborhood around the transition 
points. In our calculations, a will generally arise an overshoot /undershoot 
parameter, e.g. Equation [16]. As in PE, a is of the order of (log log r/ -1 ) 1 / 2 
and the risk can be accurately approximated in a resolution coarser than a. 
Now, note that Equation [16] reduces to (v w ^ 2 v + o) 2 = X\ + a? and so 

V V = (^/ + Vw ° 2 ) 1/2 ~ V w 2<X - "V ~ aV w 2 - 

Thus v r] G (Xj — a, Xf). So as i] — > 0, Xf is quite close to the vantage point 
Vrj (in a-coarser resolution). Also, note that the ratio Xf : A e equals Vw : 1. 
Thus, as the future variability r decreases, the distance between A/ and 
A e increases. Also as i] — > 0, the threshold behaves as A e ~ (2 log t? -1 ) 1 / 2 . 
Attributes in the asymptotic regime are pictorially represented in Figure [3] . 

7r[r], r, 2] is a sparse prior in the sense that repeated sampling from the 
prior would lead us with a sparse signal as i] — > 0. We will see that 7r [77, r, 2] 
will be an asymptotically least favorable distribution for 9. To get a fair 
understanding of our strategies in the predictive regime we formulate the 
predictive density estimation problem as a two-person game between the 
Nature and a statistician. 



2.4. Predictive Two-Player Game and Equilibrium Strategies. 

In this predictive game, Nature chooses a probability distribution tt(9) from 
m + (r]) for the location parameter 9. Then, a particular sample point $o 
is generated from tt(9) and based on the signal 9q realizations X and Y 
contaminated with white noise would be produced: X = 9q + ei and Y = 
9$ + r 1 / 2 e2 where e\ and €2 are independent. The statistician sees only X and 
he knows about the sparsity restrictions and the data generation scheme. He 
has to come up with a density estimate for Y . It is to be noted that under 
sufficient concentration properties the complicated sparse high dimensional 
minimax prediction problem is equivalent to repeated playing of this simple 
2-player game with fixed strategies (from both) over independent trials. 

As r\ — > 0, a minimax strategy of this predictive game is given by the 
positive version p[rj, T, CL + , U] of p[r], T, CL, U] where 



p[rj,T,CL + ,V](y\x) 



p(y\x;7r[ V ,r,CL + ]) if X < X 

p(y\x;nu) ifx>A 



e 



and the tt[t], r, CL + ] is a sparse discrete prior (cardinality of the support set 
equals (K„ + 2) with K v defined in Equation [17]) with (1 — 77) probability 
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Fig 3. The figure shows the support and probability allocation of the sparse 2-point prior 
7r[?7,r, 2] along with the universal threshold \ e and the ideal predictive threshold A/. The 
abscissa is graduated in a units and is drawn according to the scale with n = e" 1000 and 
r = 0.2. 

at and sharing the remaining mass on a cluster of (K v + 1) support points. 
The non-zero support points approximately lies between Aj and A e . The 
{K n + 1) non-zero support points start from [Xq = u„ with given by the 
Equation [16] and span out in a geometric progression 

Hi = (l+2r)Voj £ = 1, 2, . . . , K v and K v = max {i : (l+2r) l ^o < A e +a}. 

As 77 — > 0, a first order approximation to the cardinality would be 

log(l + r _1 ) 
21og(l + 2r)_ ' 

The subscript in would be dropped for simplicity. The non-zero support 
points are equally probable and in that way the cluster prior 

K 

Trfo, r, CL+] = (l- v ).6 + -r^— E *w (0) 

i=0 



(17) 



l0g(Ae/ A/ ) 




log(l + 2r) 
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is a probability distribution in m + (r]) which is midway between the least 
favorable prior in PE based on X and W = UMVUE(X, Y) respectively. 
The intermediation is marked by equal sharing of probability among the 
finite support points laid between A/ and A e . A schematic representation of 
this mass allocation is presented in Figure [4]. The alignment of (spacing 
between) the support points is also intrinsic to the nature of the predictive 
problem and will be discussed later (in Section 4). Note that as i] — > 0, 
tt[t], r, K] is a sparse prior and ir[q, r, CL + ] = 7r[r}, r, 2] if r > 0.42. 



1 — T] 







Af 




Ae 




v/sJ 


v/sJ 


v/sJ 





v v [il 111 



Fig 4. The figure shows the support and probability allocation of the Cluster prior 
ir[r], r, \Cl + ] along with the universal threshold A e and the ideal predictive threshold A/. 
Here with r = 2, we have 3 equally likely non-zero support points at /io = v v , Mi and /12 
which constitute a geometric progression with common ratio 1.4. The abscissa is graduated 
in a units and is drawn to the scale of n = e~ 1000 . 

Theorem 2.1. As rj — > 0, 7r[rj, r, 2] is an asymptotically least favorable 
prior for the univariate predictive game andp[rj, T, CL+, U]{y\x) is an min- 
imax estimator with the optimal asymptotic risk of (2r) _1 r] A 2 ?. 
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2.5. Proof of Theorem 1.1. As our set-up is symmetric it is enough to 
prove Theorem 2.1. We will use the the Bayes-Minimax strategy described 
before. So, we would calculate a lower bound on the Bayes risk of ir[q, r, 2] 
(see Lemma 2.1) and will produce a matching upper bound on the maximal 
risk of pt[q, T, CL + , U] (see Theorem 2.2). To interpret the result in terms of 
predictive game note that the statistician's choice for a density estimate of 
Y will involve information about 6$ stored in X as well as the Gaussianity of 
the noise distribution. And, as the parametric form of the true future density 
is known an effective density estimate will depend on efficient estimation of 
9q which in turn depends only on the sufficient statistics. In particular if 
X and Y were both observed an optimal point estimate of 9q based on the 
sufficient statistics W would produce an optimal density estimate (we will 
choose the plug-in version). So, for Nature, who aims to set the most difficult 
predictive set-up, the goal is essentially setting up the most difficult point 
estimation case for 6q based on the fact that X and Y are both observed. 
She apprehends that the statistician may produce a near accurate point 
prediction Y of Y based on X. So, the worst possible prior distribution 
involve point estimation of Oq based on observing (X, Y a )\ Let us denote 
a typical 2-point prior in m + (rj) with its only non-zero support point at v 



Here, we provide an intuitive (and a bit non-rigorous) proof of the Lemma 
by using the connections with point estimation (PE) theory. We avoid the 
intricacies of overshoot term and present asymptotic arguments in the reso- 
lution higher than the O (a). In Section 3 we have rigorous technical proofs 
of the exact asymptotic behavior of the predictive risk of any 2-point pri- 
ors in m + . For those detailed calculation, the connecting equations can not 
help much as we still need to dig into the asymptotic subtleties of the PE 
regime. Lemma 2.1 and the other 2-point priors results will follow directly 
from those calculations. 




Lemma 2.1. 



5(7r[77,7-,2])>7/-i-(l + o(l)) as v -> 



Proof. To compute the predictive Bayes risk of 7t[t), r, 2], we will use 
the Connecting equation [13] and known properties of the quadratic risk of 
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0[7T2pt[77) v]\ - the posterior mean of the 2-point prior vr2 P t[?7, v\. 
From PE theory Johnstone (2012, Section 8.4) as rj — > 0, the asymptotic 
quadratic risk q(9, #[7T2pt[?7, u]] , l) of the Bayes estimate of 7T2pt[??, v \ m an 
estimative set-up with unit noise variance, has the following properties: 

Property 1: Because of the very high mass at 0, the risk at will be 
insignificant (lower than the order of T^log??" 1 ) and the dominant pro- 
portion of the Bayes risk will be from the non-zero support point v. 

Property 2: As rj — > 0, the quadratic risk at the non-zero point v will be 
of the order of v 2 as long as v 2 < A 2 — ca\ e with c > 0. Once v 
exceeds A e the quadratic risk at v becomes negligible compared to its 
peak value. Thus, the maximal first order asymptotic quadratic risk is 
attained when v = A e and 

(a a r r ll i\ f 1,2 \i v 2 < \ 2 e - c a \ e 

q (e,e MM },i)~{ 0(1) [iu 2> xl + 2aXe ■ 

Again, we know that the Gaussian estimative set up with noise variability 
v can be reduced to an unit variance problem by suitably scaling the ob- 
servations as well as the location parameter by v 1 / 2 . Posterior probabilities 
remain invariant to the transformation leading Bayes point estimates to be 
similarly scaled. And so, the quadratic risk of Bayes estimates under the 
variance stabilizing transformation is scaled by variability v, i.e. 

q(9, e[7r 2pt [rj,u]],v) = vq(v- 1 / 2 e, e[^[r,,v-^v]] t l). 

Thus, while computing the predictive risk of the Bayes density estimate of 
7r[?7, r, 2] at is„ by the Equation [13] we have: 

P U,p[n[r),r,2]]) = j ^^v^i^lv, ^]] ,v ) dv 



i i 

—q(v~ 1/2 v v , 9 [vr2pt[?7, v~ 1/2 v v ]] , 1 ) dv. 



Also, for all v G [v w , 1] we have (v Y l 2 v^) 2 < X 2 — a\ e . So using the afore- 
mentioned Property 2, as r] — > we get 

P^,p[Av, r]] )>(l» U jdv = u 2 £ ± 2 dv = V l = | (1 + o (1)) as 

and the corresponding predictive Bayes risk satisfies 

/ \ A 2 

B(ir[i h r,2}) > V x r(v v ,p[ir[r),r]]) > ^(l + o(l)). 

This completes the proof. □ 
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Actually we can infer more about the prior ir[rj, r] and like PE, here too 
we can show that the prior 7r[r/, r] is asymptotically least favorable among 
all 2-points priors. 

Lemma 2.2. As rj — > 0, ir[r],r, 2] maximizes the asymptotic Bayes risk 
in the class of all 2-point priors in m + . 

Proof. We know (by Property 1 described in Lemma 2.1) that the risk 
at the origin will have insignificant contribution (lower than the order of 
r/Ag ) to the Bayes risk. Also, based on Property 2, for maximizing the 
risk at the non-zero support point the choice of v is reduced to the set 
{u k = k\ e : k £ [0,1]}. The predictive risk of 7T 2pt [77, i/fc] a t the non-zero 
support point will be given by, 



B (vr 2pt [77, u k ]) ~ 77 • p ( u v , p [7r 2p t [rj, u k ]) 
f 1 1 

•J Dm 



= V ^-l{ v 1,2 ^,0[iT2pt[ri,v 1/2 is k }] , 1 ) dv. 

ivt n 1 -1/2 r -1/2 w 1 \ f V ~ X k 2 Ag if v > k 2 

Now, as 77 -> 0, g(u ' !/„, [^[77, u 7 , 1 ) ~ < Q ^ . f ^ < fc2 

so, the asymptotic predictive Bayes risk is 

B(7r 2p t[77,i/ fe ]) = r j \2- 1 k 2 X 2 e [ v~ 2 dv I (l + o(l)) 

= /c 2 {l - (max^^Tj^))" 1 } 7/A 2 /2 (l + o (1)) 

which is maximized at /c = . Thus the Bayes risk is maximized for the 
2-point prior whose non-zero support point is at Ay. □ 

In this context, note that the optimal asymptotic predictive risk is always 
lower than the square error of point estimation of #0 based on X. This is 
because in the predictive set up (for Nature) the future Y could disclose 
additional information about 9$. As such, we will see afterward that the 
ratio of the optimal predictive to estimated risk is v w . For the two extreme 
cases as r approaches and 00 the ratio tends to and 1 respectively. It 
validates our intuition about this predictive set-up. As with r — > 00, even 
knowing Y will not provide any additional information about X. So, for 
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Nature the predictive problem will be as easy to set as the estimation one 
where only one sample X is explored. Similarly, as r — > 0, Y would disclose 
infinite amount of more information than X. Comparing the optimal risk 
in point estimation and the predictive regime we can say that predictive 
density estimation based on KL loss is an easier task than Point Estimation 
under quadratic loss. However, it does not say that prediction is easier than 
estimation. 

On the contrary, constructing optimal predictive densities (for the statis- 
tician) is far more complicated than designing point estimates. The issues 
that need addressing are 

Sparsity Regularization: The prior information of having at least (1 — rf) 

mass at the origin has to incorporated. 
Risk Diversification: The statistician does not see Y but can balance his 

ignorance about Y by sharing his future uncertainty. 

As such, the interplay between sparsity-regularization needs and the dy- 
namics of risk diversification (because of predictive purposes), requires to 
be explicitly tracked for calibrating optimal predictive schemes. We will see 
that the optimal strategies will lie outside the given parametric (Gaussian) 
family. However, there exists at least an asymptotic solution in the class 
of Gaussian mixture (finite) densities which is flexible enough to optimally 
balance the regularization vs diversification trade-off. 

Theorem 2.2. As r\ — > 0, for any r £ (0, oo) we have 

sup B(7r,p[rj, T, CL + , U}) < r]X 2 f /(2r)(l + o(l)). 

7rSm+(»)) 

The lemma is proved in section .4. For controlling the sparsity effect, the 
statistician can use a threshold density estimate. Threshold rules are a par- 
ticular class of non-linear estimates which may be successfully employed to 
devise sparse minimax optimal estimates particularly in location estimation. 
The idea behind threshold rules is to use an unbiased estimate (generally 
unbiased or controlled bias) when the observed data is above the threshold 
and an adjusted one if the observation is below the threshold. Here we see 
that an optimal choice of threshold can depend entirely on the degree of 
sparsity r]. As only X is observed the statistician is forced to use A e as the 
threshold. He can use the best invariant predictive density pjj if the past 
observation X crosses A e . Below the threshold, his estimate has to account 
for both sparsity effect and risk sharing. The rationale for this choice rests 
on the fact that, because of the severe sparsity constraints a near zero den- 
sity estimate is required to control the risk at the origin. If nature places the 
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entire remaining mass rj between the parametric values (0, A/], the statis- 
tician can perform under the optimal Bayes risk for this problem by using 
the zero density estimate 4>(y\0,r). However, as the supremal Bayes risk is 
\)/{2r) the zero estimator is unusable when the parametric value is greater 
than Xf. Thresholding ensures that the predictive risk above A e is bounded. 
So, the need is to control the risk in (A/, A e ] by moving away from the zero 
estimator. But, this deformation should not be large to affect the risk at the 
origin and an approach would be to use the Bayes predictive density based 
on a prior with (1 — r/) probability at the origin (this leaves the sparsity 
restrictions untouched) and with rj probability distributed in (A/,A e ]. 

2 Player Game and Equilibrium strategies 

Nature: Will choose a distribution on 9 which will make point 
estimation of 9, under quadratic loss and based on observing 
both the past X and future Y, the most difficult. 

Statistician: Will use threshold estimators. He is forced to use 
A e as threshold as he only observes X. The idea is to use 
estimator when 9 < A/ and share his risk for 9 between A / and 
A e . A predictive strategy can be constructed by using a prior 
with probability 1 — rj at and share equally the remaining 
mass r\ among a finite chain of points covering Xf and A e . 

Decision Theoretic Evaluation Game 

Through the above sharing policy we control the transition of the corre- 
sponding Bayes predictive density (and subsequently the threshold version) 
under 9 £ (0,A e ]. The Bayes predictive density will be sufficiently close to 
(j)(y\0, r) till 9 < Xj and thereafter it gradually shifts rightwards in way that 
the risk at any 9 € (Xf, X e ] is under the desired limit. The interval (A/, A e ] in- 
creases with decrease of r and the statistician is completely non-informative 
in that zone. As such (Xf, X e ] can be regarded as his most vulnerable zone. 
A way to share his predictive risk across that zone would be to divide the 
probability rj equally in a finite chain of point covering the interval. The 
non-informativeness in (Xf, X e ] is reflected in uniform sharing of the future 
uncertainty. Quantizing the vulnerability in finite locations across the inter- 
val is pivotal. As soon as the parametric value 9 crosses Xf we need sharp 
transitions from the zero-estimator for which there is need for non-zero mass 
around Xf. So, continuous sharing policies which are independent of the de- 
gree of sparsity rj will not work. 

In Section 6, we will see several efficient alignments of the chain. pr(.\x) 
is discontinuous at x = X e and the number of Gaussian mixtures in pr and 
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their weights are based on the degree of sparsity r], the future volatility 
r and the past observation X. However, neither the Bayes density density 
corresponding to vr[r/,r] attain minimax risk nor the cluster prior (which is 
the basis of pr) is least favorable. But, based on the calculation we can trace 
an infinite support prior ir[r],r, INF] on 9 which attains supremal risk and 
produces minimax Bayes density estimate. 

3. Maximal univariate Bayes risk of 2 point priors. 

Here, we will work directly with the predictive loss. Calculation will involve 
deriving closed forms for the Bayes predictive densities. In the process we will 
show that properties similar to those stated in Lemma 2.1 for the quadratic 
loss also exist for predictive densities. 

Posterior probabilities based on vT2 P t[7?, v\ is given by 

7r 2p t [V, v\(0 = 0\x) = — r— t- r— r-r and 

rj(f>{x - v) + (1 - 77) (p(x) 

7T2pt[^, V]{9 = V\ X) = 1 - 7r(0 = 0| X). 

And so, the corresponding Bayes predictive density is 



[i - V ) ■ m ■ -^( jfc) + r, ■ <t>{x - v) ■ ^A^t) 

(1 — 77) • <f>{x) + T] ■ (j){x — u) 

_J_ ( y \ (1 - rj) + rj ■ e^+vM- 1 -^^ 2 
" \Vr) (1 -n) + r ] e ux - 1 2 u2 



= (j>(y\0,r) x K(x,y). 

For doing calculations under strong sparsity, we found it most convenient to 
represent the Bayes predictive densities as tiltings of the zero-density. The 
tilt function h u (X,Y) is given by 

(18) h u (x,y) = {l + 7] (l-rj)- 1 e ux - l 2 u2 y 1 {l + V (l-7 ] )- l e^ x+y ^- 1 -^- u2 } 

with both the numerator and denominator being greater than unity which 
implies that their logarithms are always positive. 

Now, from definition we have predictive risk at as, 



where the expectation is over both X and Y which are independent Gaussian 
with common mean (denoted in subscript) and known variances 1 and r 
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respectively. Also, note that though there is a negative sign the risk is always 
positive (as we have showed before that KL divergences are always positive). 
The risk at v is given by 

v K og [pfrM](Y\x))) 

±- x E v (2vY - v 2 ) - E u {\og K(X, Y)} 
2r 

^-E v {\ogh v {X,Y)}. 
Zr 

As rj — > 0, for any 2-point prior with the very high probability (1 — if) at 
0, the loss at the origin is always small irrespective of where the non-zero 
support v is placed. We show in Lemma 3.1 that it remains bounded by rj. 
At v though, the asymptotic loss can be unbounded as r] — > 0. We will chose 
an optimal v maximizing this loss and the asymptotic maximal Bayes risk 
will be solely governed by risk at the non-zero support point in-spite of its 
low prior probability r\. 

Now as we move v away from the origin, the Bayes density estimate at 
v initially behaves like </>(.|0, r) giving rise to the first order asymptotic loss 
f 2 /2r. In these cases the tilt function h u fails to sway the predictive density 
away from the origin when the true parametric value is v. However, as we 
move v further away from 0, h y {X, Y) will be successful in tilting the pre- 
dictive density away from 0(.|O,r) and towards 4>(y\v, r). Subsequently, the 
risk at v will drop due to appreciable contribution from E v {logh u (X, Y)}. 

If the non-zero support point is (the positive root of the quadratic 
equation [16]) the tilt function is still inept enough not to cause any signif- 
icant reduction in the first order asymptotic risk at v^. The proof follows 
directly from Lemma 3.2. So, with r\ — > 0, the first order asymptotic risk 
of the Sparse 2-point prior piy^^p^^^]) > u 2 /{2r) (1 + o(l)) which in turn 
reproves Lemma 2.1 as 

B(ir[r],r,2]) = (1 - rj) X r(0,p[vr[?/,r]]) +rj X r(v v ,p[Tr[r),r]]) 

>rA(l + (l)). 
2r 

Lemma 3.1. For any r] € [0, 1) and v G [0, oo) we have, 



p( v,p[n2pt[n,v]) 
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Proof. We use the representation of h y {X, Y) given by Equation [18] and 
so can assume that the logarithm of the numerator there is non- negative. 
Hence, 

P {V t p [7T 2pt [77, U}} ) < E log {1 + 7? (1 - 7?)- V*"^ 2 } 

< v (l- r] y 1 e- /2 / 2 E e vX 

by using the inequality log(l + x) < x which holds for all non-negative x. 
Again, as X is standard normal we have Eq (exp(i/X)) = exp(- y2 /2) and we 
have the required result. □ 

Lemma 3.2. For any r\ £ [0, 1] such that a > and there exists a positive 
solution u v of Equation [16], we have 

log(l - 7?) < E v {log h u (X, Y)} < 1 + v~ 1/2 a~ 2 for all u G (0, v v \. 

Proof. We first show the upper bound. For that purpose we will use the 
representation of h u (X, Y) given by Equation [18] and so the logarithm of the 
denominator there is always positive and can be ignored while calculating 
the upper bound. We can rewrite the numerator in terms of the futuristic 
random variable W = (l+r~ l )~ l (X+Y /r) and its variance v w = (1+r -1 ) -1 
as: 

K log 1 1 + V (1 - J?)" 1 exp (v- 1 v W - X - v- 1 ^ 2 ) 



where W = N(is,v w ). We change the measure to standard normal Z 

— 1/2 

v w (W — v) and it results in 

£ log |l + 77 (1 - 77)- 1 exp (v~ 1/2 vZ+^ v- l vA \ 
Now, as v < u„ by Equation [16] we have, 

77 (1 - r])~ l exp ( v ~ 1/2 v Z + - v~ l v 2 J = c v exp (v~ 1/2 u (Z - a)) 



where c u is a constant and c u G [0,1]. Choosing c u = 1 we get the upper 
bound, 

E„ {log h u (X, Y)}<E log ^1 + exp (v^ 2 u(Z-a))\ 
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Now, we decompose the expectation of the random variable on R.H.S. con- 
ditioned on the event {Z > a}. When Z < a, the random variable (whose 
expectation is considered) is bounded by log 2. 

When Z > a, we use the naive bound: 'log(l + x) < 1 + logx if x > 1' for 
bounding the said random variable. Aggregating the two parts the ultimate 
bound would be 

E v {log h u (X, Y)} < log 2 • P(Z < a) + [P(Z > a) + v~ 1/2 uE{Z - a)+] 
< 1 + v' 1 ' 2 uE{Z - a)+ 

and the truncated Gaussian expectation can be exactly computed as 
E(Z - o)+ = / z<j)(z) dz-a $(a) = 0(a) - a $(a) < a~ 2 cp(a) < a' 2 \' f l 

J a 

where the first inequality uses the result (28) about Mill's Ratio. As < Af , 
for any v < u„ we have vE(Z — a)+ < a -2 . So, S v {log h v (X, Y)} is also 
bounded by 1 + u^^a" 2 . 

For lower bound we can similarly neglect the numerator of h u (X, Y) and so, 
E„ {log h u (X,Y)} > -E u \og (l + vil-v)-^- 1 ^ 
> - log ( 1 + 7/(1 - r/)- 1 e"^ 2 ^ e 1 ' 



which follows by Jensen's inequality. Noting, that X is standard Gaussian, 
the bound simplifies to — log (I+77 (1— t]) -1 ) = log(l— rf). Hence, proved. □ 

By Lemma 3.1 and Lemma 3.2 the asymptotic behavior of Bf^ptiV) v \) 
can be characterized when v varies in [0, v v ] . Next, we track p (y, p [~K2pt [Vi u ]\ ) 
(and hence B(7T2pt[T), v\) by using Lemma 3.1) when v > A/. 
We will prove that the risk remains bounded as v crosses the threshold A e 
(Lemma 3.3). In between Aj and A e we can show that the risk is decreas- 
ing (Lemma 3.4). However, the descent is gradual and there is no abrupt 
transition in the first order risk before A e . Thus, the maximal Bayes risk for 
2-point prior is attained around v = A/ and we have effectively character- 
ized the first order behavior of the risk with v varying along the positive 
axis in resolution of a units. As such, as ij — > 

" 2 /2r if V < U v 

p {v,p[^2pt[V) 1/ ]]) ~ { decreasing if A/ + (2v~ 1 ) 1 / 2 a <u<\ e -2a . 

0(1) ifi/> \ e + {2v- 1 ) 1 / 2 a 
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v ->■ 



Fig 5. The plot shows the risk p (v,p[-K2pt[ri; v]\ ) of the Bayes predictive density from the 
two point prior ~Ki v t\r], v\ at the non-zero support point v as v is moved along the positive 
axis. Xf and X e are marked by vertical lines and the horizontal gray line represents the 
first order maximal risk of X 2 f/2r. Reflecting the resolution of our asymptotic calculations 
the abscissa is ticked at multiples of a while the ordinate is marked in multiple of l/(2r) 
units to represent change in order of quadratic loss. The figure is actually drawn according 
to scale with rj = e -50 , r = 0.3 producing Xf — 4.8, A e = 10, a — 1.77. 
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We do not quantify the rate of descent of p {y,p\^2pt\"f]-, v]\ ) though it can 
be approximated from our proof of Lemma 3.4. In Figure [5], we trace the 
asymptotic risk by Monte Carlo simulation. It depicts the gradual descent 
which compared to PE regime is a contrast. 

Lemma 3.3. For any r\ £ (0, 1) such that a > we have, 
P fapfapthv]]) < (2 or)" 1 - riY 1 log 2 for all u>\ e + {2v7 UJ 1 ) 1,2 a. 

Proof. We know that p (y,p[ir 2p t[rii l ']\) = v 2 /2r - E v {\ogh v (X,Y)}. 
As before we can standardize the measure to standard Gaussian. Because of 
the logarithm the numerator and denominator of h v separates and can be 
standardized simultaneously. After standardization, we have: 

(19) 

/ ~r r m v 2 pi , f 1 + 77(1- yy 1 exp (y~ 1/2 vZ+\ „~ V) ] 
p („, p [vr 2pt [vM])= Yr -Eo log j J - - (1 _ — ^ - x _ ^ j 

(20) = ^ - E log / 1 + 6XP ^ " Z + 3 V ^ ~ ^ 



2r & \ i + e xp(^Z+ii/ 2 -iA2) 

by substituting r/(l - r/)" 1 by exp(-A^/2). Note that v > A e + (2«- 1 ) 1 / 2 a 
implies 2 A = v 2 — 2vb — Ag > where 6 = (4 v" 1 logy) 1 / 2 and hence the 
expectation above can be rewritten as 



E A(Z) where A(Z) = log 



1 + exp (v w ^ 2 v (Z + bvf)) ■ exp(A) • exp^ 2 /: 
1 + exp (y (Z + b)) ■ exp(A) 



1/2 

where Z is standard normal distribution. Again, when Z > —bvw , A(Z) > 
u 2 /2r. And -A(Z) is negative iff Z < —1/ which leads us to: 

E A(Z) >^-. P (Z> -bvf) -E (U + exp {y Z + V - ^Ag 1 ) J I[Z < -1/] j 



v 2 

> — • $((41ogi/) 1/2 ) - log 2 • $(-1/) 

as the random variable within the expectation is always less than log 2. 
Eventually, we arrive at 

v 2 ~ ~ 
p{u,p[^2 V t[vA]) < — •$((41og^) 1 / 2 )+log2-$(z.) < (2ar)- 1 +? ? (l-r?)' 1 log2 

where the second inequality follows from Equation [28]. Thus, we get the 
result. □ 
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Lemma 3.4. As i] -> and v 2 G ( {A/ + (2w" 1 ) 1 / 2 a} 2 , Ag - 2aA e ) the 
risk p [y,pW2pt\j]i is dominated by a decreasing function of v which is 
bounded above by X 2 /(2r). 

Proof. The risk is given by Equation [19]. Similarly as before we can 
show that as long as v 2 <X\ — 2a\ e the contribution from the denominator 
is insignificant 

E log 1 1 + exp (uZ+ l -v 2 - h 2 e ) | < O(l) 
and whenever v > A/ + (2v~ 1 ) 1 ^ 2 a we have, 

E log |l + exp (v- 1 / 2 vZ + \ v- l u 2 - ^X 2 e ) \ > \ [X 2 e - v~ l v 2 ) ■ ${V2a) . 
So, ultimately we will have 

which is decreasing in v and bounded above by X 2 /2r. □ 

It will be seen that to prove Theorem we only need Lemma 2.1 for the 
lower bound. In that regard, the extensive calculations in this section may 
seem to be an expensive regression from the objective. However, these results 
not only provide more intuition about the predictive regime but will also 
help to follow the risk calculations in the next section where the risk of the 
density estimate from a multi (K) point prior compounded with thresholding 
complications is evaluated. 

4. Minimax upper bound. 

To simplify notations the univariate strategy p[rj, T, CL + ,U] and the cluster 
prior p[r], r, CL + ] will be abbreviated as px and n[r),r,K] through out this 
section. K is the number of support point in the cluster prior and is a func- 
tion of rj and r. As mentioned before, the pr is governed by 2 major effects: 
thresholding and risk diversification. The risk diversification procedure in- 
volves (a) probability allocation (b) alignment of non-zero support points 
across the vulnerable zone, which in our pr is characterized by ir[r), r, K\. 
In this section, the risk calculations of j>t are carried out in a manner such 
that they can be easily generalized for other reasonable discrete probability 
sharing schemes (with 1 — n probability at the origin). In particular, we in- 
corporate the peculiar alignment of the non-zero support points of ir[rj, r, K] 
only toward the end of the section and the results before Lemma 4.4 will be 
reused in Section 6 to display other feasible sharing schemes. 
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Proof of Theorem 2.2. We characterize the Bayes predictive density for 
the Cluster prior. Using Bayes formula, the posterior distribution of ir[rj, r, K] 
is given by: 

r -i , | i (1 — 77) 6 (x) 

ir[ V ,r,K](0\x) = - / .14 



(1 - t?) (x) + n/(K+i) Ef=o 0= - Hi) 

V/{K+1) 4> (X - ft) 

(1 - 77) (x) + i/{k+i) Eto ^i x ~ Hi) 



,[r ) ,r,K]( H \x) = { !Z J.K ,, ;> J = 0,...,K 



and the predictive density p[vr[r7, r, K] ] (y\x) is given by 



A 



7t[r),r,K](0\x) • <p(y\0,r) + ^2ir[rj,r,K](0 \x) • <p(y\m,r) 



i=0 



= (1 - ??) 00*0 0(y|o, + E£o 0(g - r ) 

(1 - r/) 0(a) + E l =o 00 s - Hi) 

which like the 2-point prior case can be rewritten as a tilt function acting 
on the zero density ^[^[77, r, K]] = (j)(y\0, r) x ^.[^[77, r, K\ ] (x, y) where 

(21) 

h r r , 1 + Ef = o exp {*(x + |) - |(1 + r^j} 
ft [ 7r[7/, r, A J J (x, y) = z - 1 _ , t — . 

1 + K+i 52i=o ex v{Hi%- 2$} 
Observing X = x, its predictive loss at the point 6 is expressed as : 

L(»,f W ,,r,*]] (.|* - .)) - E 9 [log ( ^™W\*) 1 

= ^-Ee(h[ir[ti,r,K\](x,Y)\ 

where the discounted location distance 6 2 /2r appears due to the predictive 
loss between <j>{Y\ 9,r) and 4>(Y\ 0,r). 

We would need to study the behavior of the conditional expectation as well 

as the unconditional Kg f h\_ir[r), r, K] ] (X, Y)j (^ ne expectation is over 

both X and Y) in details. For that purpose it will be helpful to change 
the measure to central Gaussian with X and Y having variances 1 and r 
respectively 

E e (h [ Trfo, r, K] ] (X, Y)) = E (N e (X, Y) - Dg{X)) 
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where Ng and Dg are the logarithms of the numerator and denominator of 
h[7r[r],r, K] ] and 



N e (x,y) = log 
= log 



1 + 



1 + 



77(1-77) 1 



K 



K + 



7/(1 - 7?) 1 



i=0 
K 



exp <Hi[x + 



K + 



exp <v w x [ fiiW - -/if + mt 



i=0 



where is the semi-futuristic random variable and W ~ N(0, 1). Now, 
using the fact that 7/(1 — 77) ~ 1 actually equals exp(u~ 1 Aj/2) (by Equa- 
tion [16]) we have, 



(22a) 

Ne(x,y) = log 

(22b) 



1 + 1>2 exp I"* 1 (vi w - + - \>?fj I 



log 



K 



x exp {v w x iii{W + a)} 



i=0 



(22c) 



where Ti{8) = \x$ — — a^i 



A 



Thus when v w l ^Li{W + a) > for all i 6 {0, 1, . . . , K} a very naive lower 
bound is 



(22d) JV fl (x,y) > v^Ttf) - \og(K + 1) where T(6) = max 1^(0). 

i=0 

Similarly, for the denominator we have 



(22e) D e (x) = log 



77(1 - ??) 1 



K 



7/1 -L — //I ^— \ |- 1 9 /il 

1 + K + 1 exp ~ 9 Mi + W J 

i=0 



Next we calculate the risk of our threshold estimate 

Fiyyi ' \ p(y\x;nu) if X > A e 

We will later see that the threshold estimator pr(.\x) is discontinuous at 
x = A e . However, modifications of pu can be used to incorporate continuity 
correction. We are interested in finding the maximum risk of pr. However, 



3G 



due to high prior probability concentration at we have to treat the risk at 
the origin separately. Depending on X the loss of the threshold estimate is 
given by: 

L (e,p[n[ri, r, K]] (.\x)j = ^ - K N e {x -9,Y) + D e (x -9) if x < A 

L(0,p[iru] (.|*)) = \{ log(l + r" 1 ) - (1 + r)" 1 ) + |^ if z > A 

where the loss of p Vu follows from the risk calculations of linear estimators 
(see Appendix). Now, averaging over the observed data X, the risk of pt 
will be given by: 

P (e,p T ) = PB (e) + PA (9) 

where pa{9) is the risk when X crosses the threshold 

Pa{0) = -( log(l + r )-(l + r) ) P e (X > A e ) + 2fl+r) 

and the component of risk from below the threshold is 

Pb(0) = —\pb,i(6) ~ PbM 6 ) + Pb,3(0)] where, 
PbM = 9 2 <5>{\ e -9) 
p B , 2 (9) = 2rE [N e (X,Y)I {x < Xe _ 0} ] 
PB)3 (9) = 2rK [D e {X)I {x < Xe _ g} ] . 

As we have mentioned before, due to the very high probability of the pa- 
rameter to concentrate at the origin we need to bound both pa{0) and 
Ab(0) with high precision. Note that, by definition Ng(X,Y) > and 
so pu(0) < E,qDq(X) I{x<\ e } which again equals 7] + o(rj) as r\ — > by 
Lemma 4.1. Though /5_b(0) will be significantly larger than p A (0), it will not 
be enough to carry the risk at above the maximum value as, 

(23) PA (0) <\ log (1 + r" 1 ) $(A e ) + ^-^E {X 2 I {X > X<:} ) 

(24) <I A og !^l.|,( Ae ) + _l_ A0(Ae 



e 



(25) = 0(r]X) [as 0(A e ) = 77/V2vr] 

And, from calculation involving the risk of hard threshold estimators we 
know that E (X 2 I {x >x e }) = K<f>{K) Johnstone (2012, Equation (8.15)) 
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and the using the result in equation [28] we have /9b (0) = 0(r]X). Hence the 
risk at stays well below the maximal value. 

Next we need to produce an upper bound on the maximum risk at any 
non-zero parametric point. While working with r(0,pr) we saw that the 
significant contribution came from r^(0). Outside the origin, the maximal 
predictive risk is governed solely by Pb(9) which can be unbounded as rj — > 
while Pa{6) remains bounded by 2 _1 log(l + r _1 ) + (1 + r) _1 . 
Now we trace the behavior of Pb{9) as 9 varies. It will vividly demonstrate 
how the dynamics of sharing future risk can be co-ordinated with sparsity 
prior information. 




Fig 6. Schematic Diagram of the Quadratic Risk in Minimax Sparse Point Estimation. 

Pb,i(&) is the dominant portion of quadratic risk of the Hard threshold 
point estimator of 9. From point estimation theory we know that it behaves 
as 9 2 until the threshold A e and then shrinks to with a steep decent (see 
Figure [6]). ps,2 — Pb,3 is the diversification or aggregation effect. ps,3 be- 
ing based on X entirely will be insignificant before A e due to sparsity and 
negligible thereafter due to thresholding effect. It is technically proved in 
Lemma 4.1. So, ps, 2 portrays the diversification effect. It is dormant before 
Xf . In between A/ + a and A e , p\ produces significant contribution and is 
effective in bringing the predictive risk r(9,px) below A^/2r. The technical 
details of the following first order behavior of ps is carried out in a serially 
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in the Lemma 4.2, Lemma 4.3 and Lemma 4.4: 

rm { e 2 iie<\ 
pbM~[ o i{e > X + a 

Ps,2v S q otherwise 
And, p B ,i (0) - 2flr(0) < ^ for 9 e [/xq + a, A + a] . 



So we have, 



\2 



sup r(0,pr) < tt" + °(/ i o) where a = \J 2 log ^ 



and the minimax predictive risk of pr is 

sup [ r(e,p T )7r(9)d6 < {I- V )r{6,p T )+ r] sup PT (fi,\) < rAl + o (1)). 
vrem+O?) •/ e>o ^ r 

Figure [6] and Figure [1] show schematic diagrams of the univariate risk plot 
of threshold rules in the PE and density estimation framework. The sole 
difference between the two regimes is the reduction pB,2(d) m the predictive 
risk in the vulnerable zone [Aj,A e ]. Plug-in density estimates fail to attain 
this risk reduction and have risk properties like optimal threshold estimates 
in PE. To obtain the risk reduction we need to have diversified predictive 
schemes. Lemma 4.3 provides a crude lower bound on the decrement and 
Lemma 4.4 shows that it is sufficient enough to attain first order optimality. 
Figure [7] contains Monte-Carlo simulation of the different predictive risk 
curves. 

Lemma 4.1. For any r/ £ (0, 1) such that A e is well defined and greater 
than 1, we have 



(a) E {D {X)I[X <X e ]} < -log(l -77) 

(b) E {D e (X)I[X <\ e -9]}<log2 
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Proof. The first inequality follows directly from Jensen's inequality, 

K 

| / 1 , 7H1-77J - 



E {D (X)I[X< A e ]}<log 



i=0 J 



< log 



K + 

K 



i=0 



K + 



= log (1 + 77 (1 - 77)- 1 ) = - log(l - 77) 
For the second inequality, as Dg(X) is an increasing function of X we have 
E {D e {X) I[X <X e -9}}< D e (X e - 9) S(A - 9) where, 



log 1 + 



77(1-77) 1 



K 



K + 



i=0 



Also, as for each i £ {0, 1, . . . ,K} we have < \n < X e + a, and so the 
maximum value of p,iX e — m 2 /2 is at most A 2 /2 for all i 6 {0, 1, . . . , K} which 
would imply that Dq(X — 9) < log (l + 77(1 — rj)^ 1 exp( A e/2)) < log 2. Hence, 
proved. □ 

Lemma 4.2. For any 9 > A e + a, Pb,i{9) < 0(Xf) as rj — > 0. 

Proof. This Lemma follows from the risk calculations of threshold point 
estimates. Taking derivative, we see 

p' Bl {9) = 2#$(A e -9) - 9 2 (f>(X e - 9) and so for t > 

p' B>1 (X e + t) = 2(A e + t)$(t) - (A e + tf(j){t) < (2t" x - (A e + tj) (A e + t) <t>(t) 

where the inequality follows from Equation [28]. Hence, for all t > 2AJ 1 , 
Pb i(^e + t) is negative. As, 77 — > 0, we have a > 2 A~ x which implies that 
pi(9) is a decreasing function of 9 as 9 > A e + a. So, as 77 — >• 0, 

sup p B ,i{9) < (A e + a) 2 <l(a) < (A + a) 2 a" V(a) = (A + a) 2 a~ 1 A e ' 1 = 0(A/). 

0>A e +a 

□ 



Lemma 4.3. As rj — > and /or £ [z/ r/ + a, A e + a), 

PB,2(9) > 2(1 + r) T(6I) g(6») - 2r log(A" + 1) 10/iere, 



K 



1 2 

2* 



1 



X' 



T(9) = max I nfl - -ft - am - -* f 
q {9) = $(A e - 9) - 5(a) - ^ar" 1 / 2 ). 



and 
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Fig 7. Plot of the dominant portion of the predictive risk Pd{8) as 9 varies over the 
positive axis. In red, green and blue are respectively the risks of the optimal hard threshold 
plug-in estimator, unshared prediction scheme p[r, T,ir[r),r,2], U\ and the minimax optimal 
density estimate p[r, T, CL + , U\. Here, r = 0.25 , r) = e" 20 , X f = 2.83 and A e = 6.32. The 
red boxes at 2.83 and 4.24 in the yellow bar zone show the non-zero support point of the 
cluster prior ir[r), r, CL + ]. 



In particular when 9 € (A, A + a), the bound shown in the Lemma can be 
negative which certainly proves that its crudeness as we already know that 
Pb,2 is always non-negative. However, the bound is intentionally kept crude 
as it helped to increase clarity in some of the later proofs. 

Proof. Using Inequality (22d) and the fact that Ng(X, Y) is non- negative, 
we get the following lower bound: 

eJn (X,Y) ELY <\ e -0}\- \og{K + 1) 

> v~ l T{9) x P (X < A e - 6 and v~ l m{W + a) > for % = 0, 1, . . . , K) 
= v w 1 T(9) x P (X < A e -0 and W > -a) 
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as each pi is positive and we the bound the probability by 
Po{X <X e -0, W> -a) 

> Po{ - a < X < X e - and X + Yr^ 1 > -a(l + r -1 )) 

> Po{ ~ a < X < A e - 6 and Y > -a) 
= Po{ ~ a < X < X e - 9) P (Y > -a) 
= ($(A e - 0) - 9(a)) • (1 - $(ar-V 2 )) 

> $(A e - 9) - i(a) - $(ar" 1/2 ). 

Now, noting that p B ,2{0) = 2rE ^N e (X,Y)I[X < A e - 0] J the result fol- 
lows. □ 

Lemma 4.4. For 6 e [A/ + a, A e + a), Pb,i(Q) - PbA ) ^ A / + 

Proof. From Lemma 4.3, pb,i{@) — Pb,2(G) equals 

9 2 ($(a) + $(ar^ 1 / 2 )} + {0 2 - 2(1 + r) T{6)} q(9) + 2r \og{K + 1). 

In the similar way as in Lemma 4.2, we can show that as p — > 0, for all 
9 < A e + a, 9 2 $(a) = 0(X f ) and 9 2 ^(ar~ 1 / 2 ) = o(X)). 

And we show that the second sum involving 6 2 — 2(1 + r) T(9) is bounded 
by X 2 (1 + o(l)) when 9 € [Aj + a, A e + a). For this purpose, note that 

# 2 - 2(1 + r)T(9) - X 2 f = min {/•(#) + 2(1 + r)api} 

K 

< 2{1 + r) a (X e + a) + mm fi(6) 

where fi{6) = 2 - 2(1 + r)pi0 + (1 + r)p 2 + rA 2 ,. 

By construction of the cluster prior 7r[p, K, r] the points pi were geomet- 
rically starting from po = v v and with pi + \ = (1 + 2r)pi for all i £ 
{0, . . . , K — 1}. Also, the points end before A e + a. We have not used the 
properties of aligning rule anywhere before in our proof. A discrete, equi- 
probable prior distribute was all we utilized in the proofs before this stage. 
Now, we will use the properties of pi. Note that for each i £ {0, . . . , K}: 

• fi is convex in 9. 

• fi(pi) = tf-2(l + r)p 2 +(l + r)p 2 + rX 2 = -r(p 2 - X 2 ) < -r(v 2 -X 2 ) 
as pi are increasing. 
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• Define Hk+i = (1 + 2r)/ix- Then by choice of K, fiR+i > A e + a. Also, 
/iGui+i) = /if +1 - 2(1 + r)/ii/i i+ i + (1 + r)//f + rA 2 , 
= (/Xi+i - (1 + r)^) 2 - (1 + r)r/x? + rA 2 , 

and using the common ration of the geometric progression, we have 

fiiVi+l) = r 2 tf - (1 + r)rtf + r\) = -r(tf - A 2 ) < -r(^ 2 - A 2 ). 

Convexity of fi implies that if ^ < 9 < fii + i for some i G {1, . . . , K}, then 
fi(9) < 0(Xj - z/ 2 ) = o(Xj) as by Equation [16], i/ 2 - Xj < 2vl/, 2 aX f . Hence, 
for all 6 € [Xf + a, X e + a] we have minj^ fj(9) = o(A^). This complete the 
proof. □ 

5. Multivariate predictive risk. 

In this section, we would need to construct sequence of priors as (n, s) varies. 
For notational convenience we assume s as a function of n here. It can 
easily be generalized. We consider a tractable convex collection of probability 
measures in the n-dimensional space 

M(n,s n ) = L(0) : jrPn{9i ^ 0) < s n \. 

However, M(n n ,s) contains prior whose support is not confined to ©(ti, 
We consider the sub-class A4 p (n, s n ) of all product priors in A4(n, s n ). The 
least favorable prior in A4 p (n,s n ) concentrates on Q(n,s n ) when s n — > oo 
and s n /n — > as n — > oo. 

Lemma 5.1. For any n, s and r we have R(n, s, r) < nf3(s/n, r). 

Proof. The set M(n, s) contains all Dirac priors (5# V 9 £ Q(n, s) and is 
convex and weakly compact . So we can apply the Minimax Theorem 2.2 to 
have , 

R(n,s,r) < sup{ J B(7r) : ir £ M(n, s)} := B(M(n,s),r) 
and the result follows by Lemma A. 3. □ 

Based on the univariate least favorable 3-point prior, for each e < 1, we 
construct a sequence (in n) of prior 7r[n, s n , e, r] in Ai p (es n , r) as 

n 

7r[n, s n ,e,r](0) = ir[es/n, r, 3](0j). 
i=i 
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The extension into the multivariate Bayes-Minimax set up can be conducted 
through the following lemma which is proved in the Appendix. 

Lemma 5.2. As n — > oo, s — > oo then if for each e < 1 there exists an 
exchangeable product prior ir n ^(6) = Y\i=i 1T 'i-[ s / n ^ r K^?) * n -M(n,s n ) satis- 
fying the following conditions: 

a. B(ir nte ) > eB(r,M(n,es n )) 

b. 7r ft)6 (e(n,a)) -> 1 

c - fec(n,s) KnAO) P{9'Pt n ,e) de = o(B(r,M(n,s n ))) where t n>e = Tr n>e (-\@(n, 
Then, 

R(n, s, r) ~ B(r,M(n, s n )). 



Proof of Theorem 1.2. We check the conditions of the lemma for our 
least favorable prior. Consider the random variable N n which is the number 
of non-zero coordinates in a random sample from n[n, s n ,e,r]. So, N n ~ 
Binomial(ra, s n /n). 

As s/n — > the 3-point prior is least favorable in m(es/n) and hence 
we have property (a) and Lemma A. 3 implies B(ir[n,s n ,e,r]) > ef3(es n ,r). 
Property (b) also holds as 

n[n, s n , e, r] (G c (n, es„)) = P (N n > es n ) < - ^2^ N \ 

which by Chebyshev's inequality goes to as s n — > 00. 
Now, note that the support of ir[n, s n , e, r] is given by 

S n ,e = {C ■ Ci = or ±u v and N n (() < s n } 

where is given by Equation [16] with r\ = n~ 1 es n . And, the univariate 
plug-in risk p(0,p E [0]) = 9 2 /(2r) and p(6,p E [±v v ]) = {9 ± v r ,) 2 /{2r). 

So, by convexity of the relative entropy loss function we have, 



P ( 0, p[ir[n,s n ,e, 



< sup p( 9, p E [Q 



1 

< — 
~ 2r 



i:(i=0 i-.&^O 



<r~ l {\\e\\l + N n v 2 } =2r~ 1 N n v 2 ri . 
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Now integrating over the prior tt, we have 

/ n[n, s n , e, r](0) p {0,p[n[n, s n , e,r]])d0 = 2r~ x vlE{N n - Q c n }. 

J0ee c (n,s„) 

Extending the univariate minimax problem it follows that B(r,A4(n, s n )) ~ 
(2r) _1 s n vi and so the ratio 

{J3(r,.M(n,s n ))}~M \ Tr[n,s n ,e,r](0) p(9,p[ir[n,s n ,e,r]]) dG \ 

I Jeee c (n,s n ) J 

is asymptotically equal to K[N n I{Q c (n, es n )}]/E(N n ), which converges to 
as n — > oo. The convergence is a consequence of the concentration of N n as 
N n (0) = ~EN n (1 + o(l)) which follows directly from Chebyshev's inequality. 

Thus, property (c) of the Lemma is also satisfied and niLi ^l 71 ' s n/ n , r i 3](#i) 
is the asymptotically least favorable prior and the theorem follows. □ 

6. Further Insights into Minimax strategies. 

6.1. The choice of Threshold. For having an optimal threshold density 
estimate we need to have a minimum threshold of A e . The working principle 
of threshold rule is that after the threshold zone they use an estimator with 
bounded bias and as a side-effect it has considerable loss at the origin. So, 
it needs ideal calibration of the threshold as the higher thresholds decreases 
the risk contribution from above the threshold pa at the origin. 

Based on the calculations in Section 4 it follows that we need to restrict 
Pa(0) below the minimax risk r]Xj/2r. If the threshold is t = q\ e , where < 
q < 1, then from the calculations in Equation [23] we have 

PA (0) = 0{td>{t)) = 0{te- t2 ' 2 ) = 0(/) » 0( V X}). 

So, A e is the minimal threshold that is to be used. This proves Lemma 1.3. 
Note that the fact that the threshold is controlled entirely by the degree of 
sparsity resonates with general philosophy of this sparse predictive regime 
where the order of the optimal risk depends on sparsity and is scaled by 
future uncertainty. 

6.2. Sub-optimality of £,£ and Q. Based on the minimax risk of sparse 
estimation of the normal mean and the calculations in the Appendix A. 1.2, 
we see that Plug-in density estimates and in general the class of Gaussian 
predictive densities is minimax sub-optimal with the sub-optimality ratio 
being independent of r\. 
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Lemma 6.1. For any fixed r G (0,oo), under the condition of Theo- 
rem 1.2, we have 

R(n,s,r,£) ~ (1 + r" 1 ) R(n, s, r). 

Also, minimax optimal density estimates lie outside Q. Lemma 1.2 follows 
by using Theorem 1.2 and the following lemma from Mukherjee and Johnstone 
(2012b). 

Lemma 6.2. In model M.2 asn->oo and s/n — > we have 

liminf min max p(G,p) > (1 + r) _1 s log(n/s) (l + o (1)) . 
n->oo pag n 0ee„(s) 

The sub-optimality of Linear density estimates as described in Lemma 1.1 
follows from the risk calculations in Appendix A. 1.3. 

6.3. Risk sharing schemes and efficient alignments of support points. While 
constructing the minimax threshold estimator pj* we have used the cluster 
prior ir[r),r,K] below the threshold. This choice is not unique and we can 
use other proper estimates in its place. By the proof in the Section 4, it 
follows that we can not use zero-threshold estimator as then ps will only 
require ps,i and for optimality we also need the sharing effect from pb,2- 
The proof structure outline before Lemma 4.4 goes through for any finite 
prior with (1 — rj) probability at the origin and the remaining probability r/ 
being equally allocated across finite points between Xf and X e . 
So a prior with (1 — rj) mass at the orgin and sharing the remaining mass r\ 
across the non-zero support points po, . . . , px ± will produce an optimal allo- 
cation if the alignment of these points (and the cardinality of the support) 
is such that Lemma 4.4 still holds. 

For example, instead of the cluster prior 7r[r], r, K] we choose a {K\ + 2)- 
points prior whose non-zero support points are equispaced (unlike in geo- 
metric progression) and equiprobable. Let the spacing between the points 
be s. So, po = and pi = po + i s for i £ {0, . . . , K{\ where 

K\ = maxli : po + i s < \ e + a} and so as rj — > 0, K\ ~ — . 

s 

Again if we would like to equate fi(pi + i) < —r(pf — A^) as in Lemma 4.4 
we have, 

fi(pi+l) = {pi+i - (1 + r)pif - (1 + r)rp\ + r\j 
= (s - rpif - (1 + r)rp} + rXj 
= -rpj + rXj + s 2 - 2srpi. 
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Now, we solve for the condition s/xo(s/xo — 2r//j) < 0. A solution is given by 
s = 2rXf which produces a choice of K\ = [_ (2r)~ 1 {(l + r -1 ) 1 / 2 — 1} J . 

We construct the following univariate prior with the non-zero support , 
equiprobable and equidistant support points lying in between /xq and A + a 

Trfo, r, E]9) = (1 - V ) ■ 6 (0) + -^-L £ <5 W (0) 

i=0 

' r 1 + r -iu/2_ 



where /ij = (1 + 2n)//o, * = 1, • • • ; K\ and ifi 



2r 



As r] — > 0, it will produce a first order minimax optimal predictive den- 
sity estimate for the univariate restricted Bayes-Minimax problem over the 
constrained prior space m + (?]). 

In Figure [8] we have the empirical evaluations of optimal predictive 
schemes in the asymptotic regime. Figure [9] contains the risk plots under 
moderate sparsity. 

6.4. Other Minimax Estimators. 
We consider the following non-negative analogue of 7r [77, r, INF] 

00 K 

tt[77, r, INF+](0) = (1 - rf) • 6 (6) + (1 - 77) £ £ 8i 5 W . (0) where, 

= J A e + (1 + 2r) J ^ ; i = 0,...,K and j = 1, . . . , 00 
Si = (logr/ -1 ) - * for i = 1, . . . , K and 

(1 - (logTr 1 )-^) „ n 1 

(log 77 1 - 1) 

The between cluster spacing and probability distribution on the clusters is 
motivated by the construction of second order minimax optimal point es- 
timates of the normal mean in Johnstone (1994). The with-in cluster mass 
distribution is more interesting. First note that for the univariate predictive 
density estimation problem iv[rj,r, 2] or its corresponding infinite support 
geometric version is not the Bayes optimal strategy for the statistician be- 
cause it is a prediction problem and he has to share his future risk. Also, 
neither ir[rj, r, CL + ] nor its corresponding infinite support geometric version 
is least favorable as after sharing the statistician incurs the maximum risk 
at A/ (which is expected) and the risks at the other non-zero support points 
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is appreciably lower even in first order calculations. However, as rj — > a 
geometrically decreasing discrete probability sharing scheme with common 
ratio logr/ -1 solves this problem because logr/ -1 — > oo when 77 — 5- and 
hence dampens the first-order terms in the asymptotic limit. 




Fig 8. Plot of the predictive entropy risk p(8, •) for the different univariate predictive 
schemes as the parameter 8 varies over R + . In green, blue, brown and black are respectively 
the risks ofp[r, T, ir[r], r, 2], U], p[r, T, CL + , U], p[r, T, -K[n, r, E], U] and that of the Bayes 
predictive density estimate based on the infinite support prior n[n,r, INF]. Here, r = 0.25, 
77 = e~ 20 , \f — 2.83 and A e = 6.32. The brown boxes at 2.83 and 4.24 in the yellow 
zone show the non-zero support point of the cluster prior n[r),r, CL + ] and the black circles 
denote the non-zero support points ofir[ri, r, E]. 



Proof of Theorem 1.3. We prove the result in the corresponding uni- 
variate model M.2(l, i], r) with a v = 1 and the parameter space restricted 
to the non-negative axis. The risk of p(y\x, 7r [77, r, INF + ]) at the point 9 is 
given by: 

P(0) = ^(8 2 - 2rE (N^(X,Y)) + 2rE (D' e (X))) where, 
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K= log 



1 + ElrTE ex p + 



y\ 1 + r 



j=0 
oo 



i=0 
A* 



2r ^ 



1 + r 



- + 1 z 

j=0 i=o 



Now, p(0) = ?? + 0(77) as A^(X, F) > and 



E (D (Z)) < log 



00 K 



i=o i=l 

Next we note that, with probability 1 we will have, 



log(l - 77) = 77 + 0(r/ 2 ). 



£ (^(X, F)) > . max^ Uy0 - ^ - afHj - ±(j + 1)/^ (l + ^ 



j'=0,...,oo 

- log(K + 1) + O (1) and 



E (D' e (X)) < i maX K ^6 - ^ + - i(j + 1)A 2 ) + O (1). 

j=0,...,oo 

Also, the optimum value of j and i for both the numerator and denominator 
is same. And so it follows 



0(6) < — max (6 2 
Hy ' ~ 2r i=i,...,K y 

j=0,...,oo 

< — max (9 2 

2r i=i,...,K 

j=0,...,oo 



2mj 9 + f^j + 4afMj ) + O (1) 



2 IMj f + o{ l il) < M o/2r(l + o(l)). 



Risk calculations similar to Section 3 will show that the Z?(7t[t7, r, INF + ]) 
attains the above lower bound. This will complete the proof. □ 



7. Discussion. The decision theoretic techniques used here can de- 
scribe the operating characteristics of a large class of predictive strategies. 
The predictive minimax theory developed here can be extended to weak l v 
balls of varying shape and sizes (Mukherjee and Johnstone, 2012a). The cal- 
culations here are first order. Second order calculations would reflect further 
relations between estimation and prediction theory. In the non-orthogonal 
model instead of r, the minimax risk would depend on the eigen structure 
of B'B(A'A)- 1 . 
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2 4 6 

^ - 




2 4 6 

Fig 9. The figure shows the risk plots for the different univariate predictive schemes under 
moderate degree of sparsity fq = 0.001,) for the two different values of the future to past 
variances: r — 0.25 (top) and r = 1. In red, green, blue, brown and black are respectively 
the risks of the optimal hard-threshold plug-in scheme, p[ r, T, n[n, r, 2], U], p[ r, T, CL + , U], 
p[r, T, 7r[?7, r, E], U] and that of the Bayes predictive density estimate based on the infinite 
support prior n[ri, r, INF] . 



APPENDIX A 

A.l. Basic Decision Theory Results in M.2. 

As the true density <ft(-\6, a'j) in M.2(n, s, a e , cr/) is bounded above by c n = 
we can restrict the action set A n comprising of all densities in 
M. n to the set of all c n bounded densities 

A(n, Cn) = \ p : R n -)• R such that / p (y) dy = 1 and p G [0, c„] \ . 

Lemma A.l. For any p £ A n but not in A(n, c n ) there exists pb G 
A(n, c n ) that dominates p in the sense L(6,pi,) < L(9,p) for all 6 £ R n . 
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Details of the proof can be found in Brown, George and Xu (2008, Lemma2). 
For any positive density p G A n but not in A(n, c n ) the general idea for 
constructing a better estimate pt G A(n, c n ) is to truncate p on the set 
S P = {y G ]R n : p(y) > c n } and to lift it on S£, i.e. 



As the KL loss for estimators in A(n, c n ) is always defined (can be infinite 
though), they are mathematically more convenient for risk calculations than 
estimator in A n \ A(n, c n ). In the light of Lemma A.l, with out any loss of 
generality we restrict ourselves to estimators in A(n, c n ) only. Next we show 
that the Bayes predictive density defined in equation (4) actually minimizes 
the integrated Bayes risk for any prior tt in collection V(W l ) of all probability 
measures on M. n . 

Lemma A. 2. For any prior n G V(W l ) if B(ir,p n ) < oo then we have 



Proof. As the true density <p(-\6,a'j) is bounded above, the marginal 
density m n (x) < oo almost surely for all x G M n . Thus, p n is defined almost 
everywhere and by Lemma A.l, with out loss of generality we can assume 
that p G A(n, c n ). The difference in the integrated Bayes risk between any 
estimator p and the Bayes estimator is, 



as we can interchange the order of integrals by Fubini's theorem. Also, 
m 7r (x)p 7r (y|x) = f 0(x|0, <Jp) 4>{y\0, cr?) tt{6) dO , so we have, 



which is the KL divergence between the densities m.n-(x) x p^(y|x) and 
m 7r (x) xp(y|x) and so it always non-negative. This completes the proof. □ 

A. 1.1. Minimax Theorem. 
We consider the Gaussian predictive sequence model 




B{-k : p-k) < B(ir,p) for anype A 





P7r(y|x) 

p(yl x ) 



dy cbc 



(26) 



Xi = 6i + a p €i ti and yi = 9i + a p e 2 ,; 
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for i G I C N, with e\ i and €%,i are i.i.d. iV(0, 1) random variables. The 
parameter space is a collection of 6 for which ^ < oo and is denoted by 
^2(N). The action set is given by 

Aoo = \ p : R°° ->• R such that / p(y) dy = 1 and p(y) > for all y \ . 

For each n G N consider all sub-probabilities in R n bounded by c n by 
extending .A(n,c n ) to its closure 

A(n, c n ) = \ p : R n ->■ R such that / p (y) dy < 1 and p G [0, c n ] I . 

.4(n,c n ) is a sub-set of the Banach space £oo(]R n ,K) - all bounded func- 
tional in M. n . We consider the topology on A(n,c n ) induced by the weak* 
topology on £oo(IR n ,]R). 

In the predictive sequence model consider the experiment (fl, B, {Ve ■ 
6 G ^(N)}) where the sample space VL = {®i£n{ x i->Vi) '■ x i-,Vi G R} and B 
is the associated Borel sigma field. As the parameter space is £2, we have a 
dominated experiment here with 

'^{(^x + r^-^l + r- 1 )^!! 2 ) }". 

Also, for each n G N, the restricted, closed action set A(n, c n ) is weak* 
compact and the loss function L(0,p) is 

• strictly convex in p G A(n, c n , +) for any 9 G R n where A(n, c n , +) = 
{p G A(n, Cn) such that L(0,p) < 00 for all 6 G W 1 } and 

• lower semi-continuous in p on *4(n, c n ) for any fixed G R n . 

So, following the lines of Brown (1974) and Johnstone (2012, Appendix 1) 
the following minimax theorem can be attained in the predictive setting. 

Theorem A.l. Consider the predictive density estimation problem in 
the Gaussian predictive sequence model (26) with the parameter space ^(N). 
For any convex set A4 of probability measures on ^(N) we have 

inf supi?(7r,p) = sup inf B(n,p) 

Next, we describe the predictive risk of the different sub-class of estimators 
in model M.2 with a p = 1 and at = r. 



dV 

= ex P 

tfP 
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A. 1.2. Plug-in Risk. 
The risk of the plug- in predictive density pe[0] = 4>(y \ (X) , r) is given by 



p(e, <t>{-\0,r) \ = E ( 



log 



<P(Y 


0,r) 


<f>(Y 


e,r) 



(2r) _1 Eg ( — 1| Y - 0|| 2 + || Y - 0(X) 



where the expectation is over 0(x| #,r) x <^(y| #,r) - the joint density of 
(X, Y) under the true location 0. On expanding the second term in the 
above sum, the crossproduct term will vanish due to time-invariance and so 
we have 



(27) 



p(e,d>( -\e,r)^ = (2r)- l E e \\0(X) - Of = (2r)- l q(e,e,l) 



Thus, the plug-in KL predictive risk is the quadratic location risk discounted 
by the future variability, and so the risk properties of plug-in density esti- 
mates follow directly from point estimation theory. Plug-in densities are also 
called estimative densities and the plug-in predictive risk will sometimes be 
also denoted by Pe(6i 

A. 1.3. Risk of Linear predictive density estimates. 

This class consists of Bayes density estimates based on conjugate product 
normal priors Y\7=i ■ 1^' ^) wnere h-> * = 1, 2, . . . , n are the non-negative 
prior variances. They are referred to as "Linear" predictive densities because 
by Diaconis and Ylvisaker (1979) they are analogous to linear diagonal point 
estimates in the quadratic error setting. 

We exhibit the calculation for the univariate case. As the normal prior 
N(0,l) is conjugate, the posterior is also normal. Setting a = (1 + / _1 ) _1 , 
we have 

tt(9 I x) oc <f>i(x — 9 | 0, 1) x 4>i(9\ 0, 1) oc 4>\{9 \ ax, a) ~ N(ax, a) 
The corresponding predictive density is a convolution of Gaussians 

Pi(y\x) = J (f>i(y-9\0,r)(f)i(9-ax\0,a)d9 

and so is also Gaussian: pi{y\x) ~ N(ax, r + a). 
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And given the past x, its loss is given by: 

L(9,pi(. \ X = x))= Eglog<f>(Y -0\0, r) - E e log(j)(Y - ax\0, r + a) 
If, , n , E e (Y-9) 2 \ 1/ , , ,v E e (Y-ax) 2 \ 
= ~ 2 ( l0g(27Fr) + r J + 2 ( ^ + + 2 (r + a) j 



1, / a\ E (y-ax) 2 1 
- log 1 + - + 



r y 2 (r + a) 2 

and by Bias- Variance decomposition, we have, Eg (Y — ax) 2 = (9 — ax) 2 +r. 
To evaluate the risk we take expectation over the past X and again by Bias- 
Variance decomposition Eg (Y — aX) 2 = (1 — a) 2 9 2 + a 2 + r. So, the risk 
is 

t n - \ In «\ (1 _ «) 2 # 2 - a(l - a) 



2 & V r/ 2(r + a) 

As Z — 7- oo, a — > 1, we get the uniform prior Bayes Predictive Density 
(pi/). It is the best invariant density as well as minimax in the unrestricted 
parametric space (will be referred as 'canonical minimax estimator'). Thus: 

Pu{y | X = x) = N(x, 1 + r) and r(6,pu) = - log (l + r" 1 ) . 

Again, as I — > 0, a — >• and we get the zero density tfi( ■ | 0,r) with 9 2 /2r 
as its risk. 

Returning back to the multivariate case, the KL risk of the linear predictive 
density estimate 



Pl[oc] =Yl</>(.\oi X[i], {ai + r)),Oi = (1 + Z- 1 )~ 1 > k > 0, i = 1, . . . 

i=i 

is quadratic in the parameter 6 and is given by 

P ( e,p L [a] ) = \ y: log ( i + ^ ) + e $ - U). 

2 f—' V r / f-f 2 (r + a*) 

1=1 2=1 
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A. 1.4. Gaussian Predictive Risk. 

The loss of the density estimate pc[6, d](y|X) = YIi=i 4>(yi\@i(X-), d«(X)) is 
given by, 



h(e,p G [e,d)( • |x = x)) =J2^eiog(^ 



<!>{Yl\9i,di) _ 



1 

2 ^ 

i=l 



log (r 1 ci i ) + 



r + 



Integrating the loss over the past X, we find the risk 
1 



p(e,p G [0,d] =^E E « 



i=l 



<ii(X) 



A. 1.5. Mills ratio and Gaussian tails. 



The function M(u) = &(u)/4>(u) is called Mills Ratio. The following in- 
equalities will provide an approximation to the Mills ratio which will be 
very helpful for our calculations with truncated Gaussian random variable. 
By Johnstone (2012, Exercise 8.1): 



(28) for any u > we have 



<f>(u) 



ii 



1 



I _ < < 



</>(u) 



u 



And so for large u, which will typically be the case, the approximation 
<&(u) ~ u~ l (f>{u) is quite sharp. 

A. 2. Multivariate Minimax Risk. 

Lemma A. 3. For any fixed n, s, r we have B(r, A4(n, s n )) = n(3(s/n, r). 

Proof. For any prior ir on lR n and its marginals {7Tj : i = 1, . . . , n} we 
have p(9,Pmx7V2X...7v n ) = Ya=i P^uP^i)- So , 

B(tt)= f ir(6)p(9,p n )d0 < j Tr(0)p(0,p niX7T2X ... 7Tn ) = J2 [ ^M^Pm) de 
J J i=i 

Again, if tt G Ai(n, s n ) then n = tt\ x^x . . . 7r„ G .A/f p (n, s n ) and A^ p (n, s n ) C 
A4(n, s n ). So, B(r,A4(n, s n )) = B(r, Ai p (n, s n )) and due to decomposability 
of the Bayes risk for product priors we have 

( n n ~j 

B(r,M p {n,s n )) = sup < ^/3(T;,r) : < s n > . 



i=l 
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Now as /3(t, r) is concave function of r the supremum in the above expression 
occurs when r, = s/n\/i. This completes the proof. □ 

Note that for any e S (0, 1) the parametric space 0(n, es) as well as the 
prior space Ai(n, es n ) is equivariant in the sense 6(n, es) = e • 0(n, s) and 
M(n, es n ) = e ■ M(n,s n ). 

Proof of Lemma 5.2. From definition of Bayes risk it follows 
BM = / K n ,e(0)p( e S^)d6< 1 7r n>£ (0)p(0,pv n jd0 

v nje (0)p(0,p„ n€ )do\ ■Tr nte (Q n )+ / n nje (0)p(0,p Unt 

9(n,s) J i9 c (n,s) 

= 7r n , e (G(n,s)) J B(i/ n)e )+ / n n , e (e) p(0,p„ n e J dO 

J6»ee c (n,s) 

< 7r n!e (e(n,s))J?(n,s,r) + o (B(r,M(n,s n ))) 

as support of u n is contained in 0(n, s), so we have B(y n ^) < R(n, s, r) and 
we use property (c) on the second sum. 

Now, using Condition (b) of the lemma, we have R(n, s,r) > e B(r, M (n, es r 
o (B(r,M(n, s n ))) and the result follows by using the following Lemma A. 4. 

□ 

Lemma A. 4. For any fixed r £ (0, oo) we have 

limliminf^4^# = l. 

e -|-i n^oo B(r, M(n, s n )) 

Proof. The proof is similar to Exercise 4.7 in Johnstone (2012). □ 
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